Jenkins build is still unstable: beam_PostCommit_Java_ValidatesRunner_Spark #4008

[Apache Jenkins Server]

See 

Build failed in Jenkins: beam_PerformanceTests_Python #840

[Apache Jenkins Server]

See 


Changes:

[kirpichov] [BEAM-3083] Do not call getSchema() and getTable() on every element

[lcwik] [Beam-2500] Add S3FileSystem to SDKs/Java/IO

[lcwik] implement serializing AWS credentials provider

[lcwik] fixup! Clarify error message is received from SDK harness

--
Started by timer
[EnvInject] - Loading node environment variables.
Building remotely on beam7 (beam) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* 
 > +refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/*
 > git rev-parse origin/master^{commit} # timeout=10
Checking out Revision fec29b04b99976382993fbad79ae88ef44dad32a (origin/master)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f fec29b04b99976382993fbad79ae88ef44dad32a
Commit message: "Merge pull request #4493 from jkff/bq-get-schema"
 > git rev-list 71192743121d0b142828da870905369f15c64f15 # timeout=10
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
[EnvInject] - Executing scripts and injecting environment variables after the 
SCM step.
[EnvInject] - Injecting as environment variables the properties content 
SPARK_LOCAL_IP=127.0.0.1

[EnvInject] - Variables injected successfully.
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins6110558747536243775.sh
+ rm -rf PerfKitBenchmarker
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins3579965436926897140.sh
+ rm -rf .env
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins8464165503030377074.sh
+ virtualenv .env --system-site-packages
New python executable in 

Installing setuptools, pip, wheel...done.
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins4617725584641276885.sh
+ git clone https://github.com/GoogleCloudPlatform/PerfKitBenchmarker.git
Cloning into 'PerfKitBenchmarker'...
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins4941567669952889714.sh
+ .env/bin/pip install -r PerfKitBenchmarker/requirements.txt
Requirement already satisfied: absl-py in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 14))
Requirement already satisfied: jinja2>=2.7 in 
/usr/local/lib/python2.7/dist-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 15))
Requirement already satisfied: setuptools in ./.env/lib/python2.7/site-packages 
(from -r PerfKitBenchmarker/requirements.txt (line 16))
Requirement already satisfied: colorlog[windows]==2.6.0 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 17))
Requirement already satisfied: blinker>=1.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 18))
Requirement already satisfied: futures>=3.0.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 19))
Requirement already satisfied: PyYAML==3.12 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 20))
Requirement already satisfied: pint>=0.7 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 21))
Requirement already satisfied: numpy in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 22))
Requirement already satisfied: functools32 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 23))
Requirement already satisfied: contextlib2>=0.5.1 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 24))
Requirement already satisfied: pywinrm in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 25))
Requirement already satisfied: six in 
/home/jenkins/.local/lib/python2.7/site-packages (from absl-py->-r 
PerfKitBenchmarker/requirements.txt (line 14))
Requirement already satisfied: MarkupSafe>=0.23 in 
/usr/local/lib/python2.7/dist-packages (from jinja2>=2.7->-r 
PerfKitBenchmarker/requirements.txt (line 15))
Requirement already satisfied: colorama; extra == "windows" in 
/usr/lib/python2.7/dist-packages (from colorlog[windows]==2.6.0->-r 
PerfKitBenchmarker/requirements.txt (line 17))
Requirement already satisfied: xmltodict in 

Build failed in Jenkins: beam_PerformanceTests_AvroIOIT #65

[Apache Jenkins Server]

See 


Changes:

[kirpichov] [BEAM-3083] Do not call getSchema() and getTable() on every element

[lcwik] [Beam-2500] Add S3FileSystem to SDKs/Java/IO

[lcwik] implement serializing AWS credentials provider

[lcwik] fixup! Clarify error message is received from SDK harness

--
Started by timer
[EnvInject] - Loading node environment variables.
Building remotely on beam5 (beam) in workspace 

Cloning the remote Git repository
Cloning repository https://github.com/apache/beam.git
 > git init 
 >  # 
 > timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/*
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
 > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # 
 > timeout=10
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* 
 > +refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/*
 > git rev-parse origin/master^{commit} # timeout=10
Checking out Revision fec29b04b99976382993fbad79ae88ef44dad32a (origin/master)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f fec29b04b99976382993fbad79ae88ef44dad32a
Commit message: "Merge pull request #4493 from jkff/bq-get-schema"
 > git rev-list 71192743121d0b142828da870905369f15c64f15 # timeout=10
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
[EnvInject] - Executing scripts and injecting environment variables after the 
SCM step.
[EnvInject] - Injecting as environment variables the properties content 
SPARK_LOCAL_IP=127.0.0.1

[EnvInject] - Variables injected successfully.
[beam_PerformanceTests_AvroIOIT] $ /bin/bash -xe 
/tmp/jenkins5398637677869452084.sh
+ rm -rf PerfKitBenchmarker
[beam_PerformanceTests_AvroIOIT] $ /bin/bash -xe 
/tmp/jenkins1583591802180609658.sh
+ rm -rf .env
[beam_PerformanceTests_AvroIOIT] $ /bin/bash -xe 
/tmp/jenkins5837552664613719219.sh
+ virtualenv .env --system-site-packages
New python executable in .env/bin/python
Installing setuptools, pip...done.
[beam_PerformanceTests_AvroIOIT] $ /bin/bash -xe 
/tmp/jenkins8116521777034489908.sh
+ git clone https://github.com/GoogleCloudPlatform/PerfKitBenchmarker.git
Cloning into 'PerfKitBenchmarker'...
[beam_PerformanceTests_AvroIOIT] $ /bin/bash -xe 
/tmp/jenkins444228096724609453.sh
+ .env/bin/pip install -r PerfKitBenchmarker/requirements.txt
Downloading/unpacking absl-py (from -r PerfKitBenchmarker/requirements.txt 
(line 14))
  Running setup.py 
(path:
 egg_info for package absl-py

Requirement already satisfied (use --upgrade to upgrade): jinja2>=2.7 in 
/usr/local/lib/python2.7/dist-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 15))
Requirement already satisfied (use --upgrade to upgrade): setuptools in 
./.env/lib/python2.7/site-packages (from -r PerfKitBenchmarker/requirements.txt 
(line 16))
Downloading/unpacking colorlog[windows]==2.6.0 (from -r 
PerfKitBenchmarker/requirements.txt (line 17))
  Downloading colorlog-2.6.0-py2.py3-none-any.whl
Downloading/unpacking blinker>=1.3 (from -r PerfKitBenchmarker/requirements.txt 
(line 18))
  Running setup.py 
(path:
 egg_info for package blinker

Downloading/unpacking futures>=3.0.3 (from -r 
PerfKitBenchmarker/requirements.txt (line 19))
  Downloading futures-3.2.0-py2-none-any.whl
Requirement already satisfied (use --upgrade to upgrade): PyYAML==3.12 in 
/usr/local/lib/python2.7/dist-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 20))
Downloading/unpacking pint>=0.7 (from -r PerfKitBenchmarker/requirements.txt 
(line 21))
  Running setup.py 
(path:
 egg_info for package pint

no previously-included directories found matching 'docs/_build'
no previously-included directories found matching 'docs/_themes/.git'
warning: no previously-included files matching '*.pyc' found anywhere in 
distribution
warning: no previously-included files matching '*~' found anywhere in 
distribution
warning: no previously-included files matching '.DS_Store' found anywhere 
in distribution
warning: no previously-included files matching 

Build failed in Jenkins: beam_PerformanceTests_Spark #1284

[Apache Jenkins Server]

See 


Changes:

[kirpichov] [BEAM-3083] Do not call getSchema() and getTable() on every element

[lcwik] [Beam-2500] Add S3FileSystem to SDKs/Java/IO

[lcwik] implement serializing AWS credentials provider

[lcwik] fixup! Clarify error message is received from SDK harness

--
Started by timer
[EnvInject] - Loading node environment variables.
Building remotely on beam2 (beam) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* 
 > +refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/*
 > git rev-parse origin/master^{commit} # timeout=10
Checking out Revision fec29b04b99976382993fbad79ae88ef44dad32a (origin/master)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f fec29b04b99976382993fbad79ae88ef44dad32a
Commit message: "Merge pull request #4493 from jkff/bq-get-schema"
 > git rev-list 71192743121d0b142828da870905369f15c64f15 # timeout=10
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
[EnvInject] - Executing scripts and injecting environment variables after the 
SCM step.
[EnvInject] - Injecting as environment variables the properties content 
SPARK_LOCAL_IP=127.0.0.1

[EnvInject] - Variables injected successfully.
[beam_PerformanceTests_Spark] $ /bin/bash -xe /tmp/jenkins8357201591424437278.sh
+ rm -rf PerfKitBenchmarker
[beam_PerformanceTests_Spark] $ /bin/bash -xe /tmp/jenkins1402021936782178025.sh
+ rm -rf .env
[beam_PerformanceTests_Spark] $ /bin/bash -xe /tmp/jenkins618848476533213586.sh
+ virtualenv .env --system-site-packages
New python executable in 

Installing setuptools, pip, wheel...done.
[beam_PerformanceTests_Spark] $ /bin/bash -xe /tmp/jenkins6915344884822081226.sh
+ git clone https://github.com/GoogleCloudPlatform/PerfKitBenchmarker.git
Cloning into 'PerfKitBenchmarker'...
[beam_PerformanceTests_Spark] $ /bin/bash -xe /tmp/jenkins5694240429376595719.sh
+ .env/bin/pip install -r PerfKitBenchmarker/requirements.txt
Requirement already satisfied: absl-py in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 14))
Requirement already satisfied: jinja2>=2.7 in 
/usr/local/lib/python2.7/dist-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 15))
Requirement already satisfied: setuptools in ./.env/lib/python2.7/site-packages 
(from -r PerfKitBenchmarker/requirements.txt (line 16))
Requirement already satisfied: colorlog[windows]==2.6.0 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 17))
Requirement already satisfied: blinker>=1.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 18))
Requirement already satisfied: futures>=3.0.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 19))
Requirement already satisfied: PyYAML==3.12 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 20))
Requirement already satisfied: pint>=0.7 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 21))
Requirement already satisfied: numpy in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 22))
Requirement already satisfied: functools32 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 23))
Requirement already satisfied: contextlib2>=0.5.1 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 24))
Requirement already satisfied: pywinrm in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 25))
Requirement already satisfied: six in 
/home/jenkins/.local/lib/python2.7/site-packages (from absl-py->-r 
PerfKitBenchmarker/requirements.txt (line 14))
Requirement already satisfied: MarkupSafe in 
/usr/local/lib/python2.7/dist-packages (from jinja2>=2.7->-r 
PerfKitBenchmarker/requirements.txt (line 15))
Requirement already satisfied: colorama; extra == "windows" in 
/usr/lib/python2.7/dist-packages (from colorlog[windows]==2.6.0->-r 
PerfKitBenchmarker/requirements.txt (line 17))
Requirement already satisfied: xmltodict in 

Build failed in Jenkins: beam_PerformanceTests_Compressed_TextIOIT #64

[Apache Jenkins Server]

See 


Changes:

[kirpichov] [BEAM-3083] Do not call getSchema() and getTable() on every element

[lcwik] [Beam-2500] Add S3FileSystem to SDKs/Java/IO

[lcwik] implement serializing AWS credentials provider

[lcwik] fixup! Clarify error message is received from SDK harness

--
Started by timer
[EnvInject] - Loading node environment variables.
Building remotely on beam6 (beam) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* 
 > +refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/*
 > git rev-parse origin/master^{commit} # timeout=10
Checking out Revision fec29b04b99976382993fbad79ae88ef44dad32a (origin/master)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f fec29b04b99976382993fbad79ae88ef44dad32a
Commit message: "Merge pull request #4493 from jkff/bq-get-schema"
 > git rev-list 71192743121d0b142828da870905369f15c64f15 # timeout=10
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
[EnvInject] - Executing scripts and injecting environment variables after the 
SCM step.
[EnvInject] - Injecting as environment variables the properties content 
SPARK_LOCAL_IP=127.0.0.1

[EnvInject] - Variables injected successfully.
[beam_PerformanceTests_Compressed_TextIOIT] $ /bin/bash -xe 
/tmp/jenkins569966758761578201.sh
+ rm -rf PerfKitBenchmarker
[beam_PerformanceTests_Compressed_TextIOIT] $ /bin/bash -xe 
/tmp/jenkins79573444673553224.sh
+ rm -rf .env
[beam_PerformanceTests_Compressed_TextIOIT] $ /bin/bash -xe 
/tmp/jenkins7847902455184232520.sh
+ virtualenv .env --system-site-packages
New python executable in .env/bin/python
Installing setuptools, pip...done.
[beam_PerformanceTests_Compressed_TextIOIT] $ /bin/bash -xe 
/tmp/jenkins7807701984381150600.sh
+ git clone https://github.com/GoogleCloudPlatform/PerfKitBenchmarker.git
Cloning into 'PerfKitBenchmarker'...
[beam_PerformanceTests_Compressed_TextIOIT] $ /bin/bash -xe 
/tmp/jenkins8686181004923928040.sh
+ .env/bin/pip install -r PerfKitBenchmarker/requirements.txt
Requirement already satisfied (use --upgrade to upgrade): absl-py in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 14))
Requirement already satisfied (use --upgrade to upgrade): jinja2>=2.7 in 
/usr/local/lib/python2.7/dist-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 15))
Requirement already satisfied (use --upgrade to upgrade): setuptools in 
./.env/lib/python2.7/site-packages (from -r PerfKitBenchmarker/requirements.txt 
(line 16))
Requirement already satisfied (use --upgrade to upgrade): 
colorlog[windows]==2.6.0 in /home/jenkins/.local/lib/python2.7/site-packages 
(from -r PerfKitBenchmarker/requirements.txt (line 17))
  Installing extra requirements: 'windows'
Requirement already satisfied (use --upgrade to upgrade): blinker>=1.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 18))
Requirement already satisfied (use --upgrade to upgrade): futures>=3.0.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 19))
Requirement already satisfied (use --upgrade to upgrade): PyYAML==3.12 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 20))
Requirement already satisfied (use --upgrade to upgrade): pint>=0.7 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 21))
Requirement already satisfied (use --upgrade to upgrade): numpy in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 22))
Requirement already satisfied (use --upgrade to upgrade): functools32 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 23))
Requirement already satisfied (use --upgrade to upgrade): contextlib2>=0.5.1 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 24))
Requirement already satisfied (use --upgrade to upgrade): pywinrm in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 25))
Cleaning up...
[beam_PerformanceTests_Compressed_TextIOIT] $ /bin/bash -xe 
/tmp/jenkins5402412292852280336.sh
+ .env/bin/pip install -e 'src/sdks/python/[gcp,test]'

Build failed in Jenkins: beam_PerformanceTests_TFRecordIOIT #64

[Apache Jenkins Server]

See 


Changes:

[kirpichov] [BEAM-3083] Do not call getSchema() and getTable() on every element

[lcwik] [Beam-2500] Add S3FileSystem to SDKs/Java/IO

[lcwik] implement serializing AWS credentials provider

[lcwik] fixup! Clarify error message is received from SDK harness

--
Started by timer
[EnvInject] - Loading node environment variables.
Building remotely on beam4 (beam) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* 
 > +refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/*
 > git rev-parse origin/master^{commit} # timeout=10
Checking out Revision fec29b04b99976382993fbad79ae88ef44dad32a (origin/master)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f fec29b04b99976382993fbad79ae88ef44dad32a
Commit message: "Merge pull request #4493 from jkff/bq-get-schema"
 > git rev-list 71192743121d0b142828da870905369f15c64f15 # timeout=10
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
[EnvInject] - Executing scripts and injecting environment variables after the 
SCM step.
[EnvInject] - Injecting as environment variables the properties content 
SPARK_LOCAL_IP=127.0.0.1

[EnvInject] - Variables injected successfully.
[beam_PerformanceTests_TFRecordIOIT] $ /bin/bash -xe 
/tmp/jenkins2381654588649916919.sh
+ rm -rf PerfKitBenchmarker
[beam_PerformanceTests_TFRecordIOIT] $ /bin/bash -xe 
/tmp/jenkins2567292030197107859.sh
+ rm -rf .env
[beam_PerformanceTests_TFRecordIOIT] $ /bin/bash -xe 
/tmp/jenkins4203094548994600056.sh
+ virtualenv .env --system-site-packages
New python executable in .env/bin/python
Installing setuptools, pip...done.
[beam_PerformanceTests_TFRecordIOIT] $ /bin/bash -xe 
/tmp/jenkins8500594767357382581.sh
+ git clone https://github.com/GoogleCloudPlatform/PerfKitBenchmarker.git
Cloning into 'PerfKitBenchmarker'...
[beam_PerformanceTests_TFRecordIOIT] $ /bin/bash -xe 
/tmp/jenkins6230543332309026170.sh
+ .env/bin/pip install -r PerfKitBenchmarker/requirements.txt
Requirement already satisfied (use --upgrade to upgrade): absl-py in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 14))
Requirement already satisfied (use --upgrade to upgrade): jinja2>=2.7 in 
/usr/local/lib/python2.7/dist-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 15))
Requirement already satisfied (use --upgrade to upgrade): setuptools in 
./.env/lib/python2.7/site-packages (from -r PerfKitBenchmarker/requirements.txt 
(line 16))
Requirement already satisfied (use --upgrade to upgrade): 
colorlog[windows]==2.6.0 in /home/jenkins/.local/lib/python2.7/site-packages 
(from -r PerfKitBenchmarker/requirements.txt (line 17))
  Installing extra requirements: 'windows'
Requirement already satisfied (use --upgrade to upgrade): blinker>=1.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 18))
Requirement already satisfied (use --upgrade to upgrade): futures>=3.0.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 19))
Requirement already satisfied (use --upgrade to upgrade): PyYAML==3.12 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 20))
Requirement already satisfied (use --upgrade to upgrade): pint>=0.7 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 21))
Requirement already satisfied (use --upgrade to upgrade): numpy in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 22))
Requirement already satisfied (use --upgrade to upgrade): functools32 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 23))
Requirement already satisfied (use --upgrade to upgrade): contextlib2>=0.5.1 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 24))
Requirement already satisfied (use --upgrade to upgrade): pywinrm in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 25))
Cleaning up...
[beam_PerformanceTests_TFRecordIOIT] $ /bin/bash -xe 
/tmp/jenkins8040070145744902677.sh
+ .env/bin/pip install -e 'src/sdks/python/[gcp,test]'
Obtaining 

Build failed in Jenkins: beam_PerformanceTests_TextIOIT #72

[Apache Jenkins Server]

See 


Changes:

[kirpichov] [BEAM-3083] Do not call getSchema() and getTable() on every element

[lcwik] [Beam-2500] Add S3FileSystem to SDKs/Java/IO

[lcwik] implement serializing AWS credentials provider

[lcwik] fixup! Clarify error message is received from SDK harness

--
Started by timer
[EnvInject] - Loading node environment variables.
Building remotely on beam4 (beam) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* 
 > +refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/*
 > git rev-parse origin/master^{commit} # timeout=10
Checking out Revision fec29b04b99976382993fbad79ae88ef44dad32a (origin/master)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f fec29b04b99976382993fbad79ae88ef44dad32a
Commit message: "Merge pull request #4493 from jkff/bq-get-schema"
 > git rev-list 71192743121d0b142828da870905369f15c64f15 # timeout=10
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
[EnvInject] - Executing scripts and injecting environment variables after the 
SCM step.
[EnvInject] - Injecting as environment variables the properties content 
SPARK_LOCAL_IP=127.0.0.1

[EnvInject] - Variables injected successfully.
[beam_PerformanceTests_TextIOIT] $ /bin/bash -xe 
/tmp/jenkins8853254196174519888.sh
+ rm -rf PerfKitBenchmarker
[beam_PerformanceTests_TextIOIT] $ /bin/bash -xe 
/tmp/jenkins6869624700421194394.sh
+ rm -rf .env
[beam_PerformanceTests_TextIOIT] $ /bin/bash -xe 
/tmp/jenkins6329642730365486285.sh
+ virtualenv .env --system-site-packages
New python executable in .env/bin/python
Installing setuptools, pip...done.
[beam_PerformanceTests_TextIOIT] $ /bin/bash -xe 
/tmp/jenkins6687550704422718157.sh
+ git clone https://github.com/GoogleCloudPlatform/PerfKitBenchmarker.git
Cloning into 'PerfKitBenchmarker'...
[beam_PerformanceTests_TextIOIT] $ /bin/bash -xe 
/tmp/jenkins4704068742711554033.sh
+ .env/bin/pip install -r PerfKitBenchmarker/requirements.txt
Requirement already satisfied (use --upgrade to upgrade): absl-py in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 14))
Requirement already satisfied (use --upgrade to upgrade): jinja2>=2.7 in 
/usr/local/lib/python2.7/dist-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 15))
Requirement already satisfied (use --upgrade to upgrade): setuptools in 
./.env/lib/python2.7/site-packages (from -r PerfKitBenchmarker/requirements.txt 
(line 16))
Requirement already satisfied (use --upgrade to upgrade): 
colorlog[windows]==2.6.0 in /home/jenkins/.local/lib/python2.7/site-packages 
(from -r PerfKitBenchmarker/requirements.txt (line 17))
  Installing extra requirements: 'windows'
Requirement already satisfied (use --upgrade to upgrade): blinker>=1.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 18))
Requirement already satisfied (use --upgrade to upgrade): futures>=3.0.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 19))
Requirement already satisfied (use --upgrade to upgrade): PyYAML==3.12 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 20))
Requirement already satisfied (use --upgrade to upgrade): pint>=0.7 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 21))
Requirement already satisfied (use --upgrade to upgrade): numpy in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 22))
Requirement already satisfied (use --upgrade to upgrade): functools32 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 23))
Requirement already satisfied (use --upgrade to upgrade): contextlib2>=0.5.1 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 24))
Requirement already satisfied (use --upgrade to upgrade): pywinrm in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 25))
Cleaning up...
[beam_PerformanceTests_TextIOIT] $ /bin/bash -xe 
/tmp/jenkins5779766645236081078.sh
+ .env/bin/pip install -e 'src/sdks/python/[gcp,test]'
Obtaining 

Jenkins build is back to stable : beam_PostCommit_Java_ValidatesRunner_Dataflow #4808

[Apache Jenkins Server]

See 

Jenkins build is still unstable: beam_PostCommit_Java_ValidatesRunner_Spark #4007

[Apache Jenkins Server]

See 

Jenkins build is still unstable: beam_PostCommit_Java_ValidatesRunner_Spark #4006

[Apache Jenkins Server]

See 

[jira] [Commented] (BEAM-3106) Consider not pinning all python dependencies, or moving them to requirements.txt

[Ahmet Altay (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-3106?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16341914#comment-16341914
 ] 

Ahmet Altay commented on BEAM-3106:
---

I mentioned upgrading the libraries that beam depends on the their newer 
versions. I do not think there is a good solution today for mixing and matching 
dependencies in the same virtualenv.

> Consider not pinning all python dependencies, or moving them to 
> requirements.txt
> 
>
> Key: BEAM-3106
> URL: https://issues.apache.org/jira/browse/BEAM-3106
> Project: Beam
>  Issue Type: Wish
>  Components: build-system
>Affects Versions: 2.1.0
> Environment: python
>Reporter: Maximilian Roos
>Priority: Major
>
> Currently all python dependencies are [pinned or 
> capped|https://github.com/apache/beam/blob/master/sdks/python/setup.py#L97]
> While there's a good argument for supplying a `requirements.txt` with well 
> tested dependencies, having them specified in `setup.py` forces them to an 
> exact state on each install of Beam. This makes using Beam in any environment 
> with other libraries nigh on impossible. 
> This is particularly severe for the `gcp` dependencies, where we have 
> libraries that won't work with an older version (but Beam _does_ work with an 
> newer version). We have to do a bunch of gymnastics to get the correct 
> versions installed because of this. Unfortunately, airflow repeats this 
> practice and conflicts on a number of dependencies, adding further 
> complication (but, again there is no real conflict).
> I haven't seen this practice outside of the Apache & Google ecosystem - for 
> example no libraries in numerical python do this. Here's a [discussion on 
> SO|https://stackoverflow.com/questions/28509481/should-i-pin-my-python-dependencies-versions]



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (BEAM-3106) Consider not pinning all python dependencies, or moving them to requirements.txt

[RK (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-3106?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16341912#comment-16341912
 ] 

RK commented on BEAM-3106:
--

This can result in some difficult-to-pin-down errors in the 
google-cloud-platform python libraries. For example, on a clean virtualenv:

 
{code:java}
pip install google-cloud-storage
{code}
Now:

 

 
{code:java}
from google.cloud.storage import Client
print(Client().bucket("gcp-public-data-landsat")\

.blob("LE00/PRE/001/049/LE07_L1TP_001049_20160215_20161015_01_T1/LE07_L1TP_001049_20160215_20161015_01_T1_ANG.txt")\
.download_as_string())
{code}
Works as expected, but after installing apache_beam[gcp]

 

 
{code:java}
pip install apache_beam[gcp]
{code}
 
{code:java}
# same code as above
from google.cloud.storage import Client
print(Client().bucket("gcp-public-data-landsat")\

.blob("LE00/PRE/001/049/LE07_L1TP_001049_20160215_20161015_01_T1/LE07_L1TP_001049_20160215_20161015_01_T1_ANG.txt")\
.download_as_string())
# File 
"/Users/karbr001/Documents/tmp_gcs_test/env/lib/python2.7/site-packages/google_auth_httplib2.py",
 line 198, in request
#    uri, method, body=body, headers=request_headers, **kwargs)
# TypeError: request() got an unexpected keyword argument 'data'
{code}
[~altay] what's the temporary relief you mentioned? Is it just installing beam 
with a custom setup.py file that points bigquery 0.28?

 

 

 

 

> Consider not pinning all python dependencies, or moving them to 
> requirements.txt
> 
>
> Key: BEAM-3106
> URL: https://issues.apache.org/jira/browse/BEAM-3106
> Project: Beam
>  Issue Type: Wish
>  Components: build-system
>Affects Versions: 2.1.0
> Environment: python
>Reporter: Maximilian Roos
>Priority: Major
>
> Currently all python dependencies are [pinned or 
> capped|https://github.com/apache/beam/blob/master/sdks/python/setup.py#L97]
> While there's a good argument for supplying a `requirements.txt` with well 
> tested dependencies, having them specified in `setup.py` forces them to an 
> exact state on each install of Beam. This makes using Beam in any environment 
> with other libraries nigh on impossible. 
> This is particularly severe for the `gcp` dependencies, where we have 
> libraries that won't work with an older version (but Beam _does_ work with an 
> newer version). We have to do a bunch of gymnastics to get the correct 
> versions installed because of this. Unfortunately, airflow repeats this 
> practice and conflicts on a number of dependencies, adding further 
> complication (but, again there is no real conflict).
> I haven't seen this practice outside of the Apache & Google ecosystem - for 
> example no libraries in numerical python do this. Here's a [discussion on 
> SO|https://stackoverflow.com/questions/28509481/should-i-pin-my-python-dependencies-versions]



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Comment Edited] (BEAM-3106) Consider not pinning all python dependencies, or moving them to requirements.txt

[RK (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-3106?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16341912#comment-16341912
 ] 

RK edited comment on BEAM-3106 at 1/27/18 2:31 AM:
---

This can result in some difficult-to-pin-down errors in the 
google-cloud-platform python libraries. For example, on a clean virtualenv:
{code:java}
pip install google-cloud-storage
{code}
Now:
{code:java}
from google.cloud.storage import Client
print(Client().bucket("gcp-public-data-landsat")\

.blob("LE00/PRE/001/049/LE07_L1TP_001049_20160215_20161015_01_T1/LE07_L1TP_001049_20160215_20161015_01_T1_ANG.txt")\
.download_as_string())
{code}
Works as expected, but after installing apache_beam[gcp] 
{code:java}
pip install apache_beam[gcp]
{code}
{code:java}
# same code as above
from google.cloud.storage import Client
print(Client().bucket("gcp-public-data-landsat")\

.blob("LE00/PRE/001/049/LE07_L1TP_001049_20160215_20161015_01_T1/LE07_L1TP_001049_20160215_20161015_01_T1_ANG.txt")\
.download_as_string())
# File 
"/Users/karbr001/Documents/tmp_gcs_test/env/lib/python2.7/site-packages/google_auth_httplib2.py",
 line 198, in request
#    uri, method, body=body, headers=request_headers, **kwargs)
# TypeError: request() got an unexpected keyword argument 'data'
{code}
[~altay] what's the temporary relief you mentioned? Is it just installing beam 
with a custom setup.py file that points bigquery 0.28?

 

 

 

 


was (Author: karbr89):
This can result in some difficult-to-pin-down errors in the 
google-cloud-platform python libraries. For example, on a clean virtualenv:

 
{code:java}
pip install google-cloud-storage
{code}
Now:

 

 
{code:java}
from google.cloud.storage import Client
print(Client().bucket("gcp-public-data-landsat")\

.blob("LE00/PRE/001/049/LE07_L1TP_001049_20160215_20161015_01_T1/LE07_L1TP_001049_20160215_20161015_01_T1_ANG.txt")\
.download_as_string())
{code}
Works as expected, but after installing apache_beam[gcp]

 

 
{code:java}
pip install apache_beam[gcp]
{code}
 
{code:java}
# same code as above
from google.cloud.storage import Client
print(Client().bucket("gcp-public-data-landsat")\

.blob("LE00/PRE/001/049/LE07_L1TP_001049_20160215_20161015_01_T1/LE07_L1TP_001049_20160215_20161015_01_T1_ANG.txt")\
.download_as_string())
# File 
"/Users/karbr001/Documents/tmp_gcs_test/env/lib/python2.7/site-packages/google_auth_httplib2.py",
 line 198, in request
#    uri, method, body=body, headers=request_headers, **kwargs)
# TypeError: request() got an unexpected keyword argument 'data'
{code}
[~altay] what's the temporary relief you mentioned? Is it just installing beam 
with a custom setup.py file that points bigquery 0.28?

 

 

 

 

> Consider not pinning all python dependencies, or moving them to 
> requirements.txt
> 
>
> Key: BEAM-3106
> URL: https://issues.apache.org/jira/browse/BEAM-3106
> Project: Beam
>  Issue Type: Wish
>  Components: build-system
>Affects Versions: 2.1.0
> Environment: python
>Reporter: Maximilian Roos
>Priority: Major
>
> Currently all python dependencies are [pinned or 
> capped|https://github.com/apache/beam/blob/master/sdks/python/setup.py#L97]
> While there's a good argument for supplying a `requirements.txt` with well 
> tested dependencies, having them specified in `setup.py` forces them to an 
> exact state on each install of Beam. This makes using Beam in any environment 
> with other libraries nigh on impossible. 
> This is particularly severe for the `gcp` dependencies, where we have 
> libraries that won't work with an older version (but Beam _does_ work with an 
> newer version). We have to do a bunch of gymnastics to get the correct 
> versions installed because of this. Unfortunately, airflow repeats this 
> practice and conflicts on a number of dependencies, adding further 
> complication (but, again there is no real conflict).
> I haven't seen this practice outside of the Apache & Google ecosystem - for 
> example no libraries in numerical python do this. Here's a [discussion on 
> SO|https://stackoverflow.com/questions/28509481/should-i-pin-my-python-dependencies-versions]



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[beam] branch master updated (3cf6353 -> fec29b0)

[jkff]

This is an automated email from the ASF dual-hosted git repository.

jkff pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from 3cf6353  fixup! Clarify error message is received from SDK harness
 add fa0c691  [BEAM-3083] Do not call getSchema() and getTable() on every 
element
 new fec29b0  Merge pull request #4493 from jkff/bq-get-schema

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 .../beam/sdk/io/gcp/bigquery/CreateTables.java | 169 -
 .../sdk/io/gcp/bigquery/BigQueryIOWriteTest.java   |   2 +-
 2 files changed, 95 insertions(+), 76 deletions(-)

-- 
To stop receiving notification emails like this one, please contact
j...@apache.org.

[beam] 01/01: Merge pull request #4493 from jkff/bq-get-schema

[jkff]

This is an automated email from the ASF dual-hosted git repository.

jkff pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git

commit fec29b04b99976382993fbad79ae88ef44dad32a
Merge: 3cf6353 fa0c691
Author: Eugene Kirpichov 
AuthorDate: Fri Jan 26 18:21:26 2018 -0800

Merge pull request #4493 from jkff/bq-get-schema

[BEAM-3083] Do not call getSchema() and getTable() on every element

 .../beam/sdk/io/gcp/bigquery/CreateTables.java | 169 -
 .../sdk/io/gcp/bigquery/BigQueryIOWriteTest.java   |   2 +-
 2 files changed, 95 insertions(+), 76 deletions(-)

-- 
To stop receiving notification emails like this one, please contact
j...@apache.org.

Jenkins build is still unstable: beam_PostCommit_Java_ValidatesRunner_Spark #4005

[Apache Jenkins Server]

See 

Jenkins build became unstable: beam_PostCommit_Java_ValidatesRunner_Dataflow #4807

[Apache Jenkins Server]

See 

[beam] branch master updated: fixup! Clarify error message is received from SDK harness

[lcwik]

This is an automated email from the ASF dual-hosted git repository.

lcwik pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git


The following commit(s) were added to refs/heads/master by this push:
 new 3cf6353  fixup! Clarify error message is received from SDK harness
3cf6353 is described below

commit 3cf635322376fce15eab27137007f5938a236cdc
Author: Robert Burke 
AuthorDate: Fri Jan 26 16:51:14 2018 -0800

fixup! Clarify error message is received from SDK harness
---
 .../apache/beam/runners/fnexecution/control/FnApiControlClient.java | 6 +-
 1 file changed, 5 insertions(+), 1 deletion(-)

diff --git 
a/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/control/FnApiControlClient.java
 
b/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/control/FnApiControlClient.java
index 928dbdd..2897189 100644
--- 
a/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/control/FnApiControlClient.java
+++ 
b/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/control/FnApiControlClient.java
@@ -131,7 +131,11 @@ class FnApiControlClient implements Closeable {
 if (response.getError().isEmpty()) {
   completableFuture.set(response);
 } else {
-  completableFuture.setException(new 
RuntimeException(response.getError()));
+  completableFuture.setException(
+  new RuntimeException(String.format(
+  "Error received from SDK harness for instruction %s: %s",
+  response.getInstructionId(),
+  response.getError(;
 }
   }
 }

-- 
To stop receiving notification emails like this one, please contact
lc...@apache.org.

Jenkins build is still unstable: beam_PostCommit_Java_ValidatesRunner_Spark #4004

[Apache Jenkins Server]

See 

Build failed in Jenkins: beam_PerformanceTests_Spark #1283

[Apache Jenkins Server]

See 


Changes:

[zoy] Increasing BatchElements's max_batch_size to 10K

[lcwik] [BEAM-2273] Cleanup examples Maven Archetype to copy in a clean state

[XuMingmin] [BEAM-3526] KakfaIO support for finalizeCheckpoint() (#4481)

[kedin] Add Avro dependency to KafkaIO

--
Started by timer
[EnvInject] - Loading node environment variables.
Building remotely on beam2 (beam) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* 
 > +refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/*
 > git rev-parse origin/master^{commit} # timeout=10
Checking out Revision 71192743121d0b142828da870905369f15c64f15 (origin/master)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 71192743121d0b142828da870905369f15c64f15
Commit message: "Merge pull request #4503: Add Avro dependency to KafkaIO"
 > git rev-list d4a55d9f485a06a9fc0933884c36d38af3c69d3d # timeout=10
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
[EnvInject] - Executing scripts and injecting environment variables after the 
SCM step.
[EnvInject] - Injecting as environment variables the properties content 
SPARK_LOCAL_IP=127.0.0.1

[EnvInject] - Variables injected successfully.
[beam_PerformanceTests_Spark] $ /bin/bash -xe /tmp/jenkins3548088816705790413.sh
+ rm -rf PerfKitBenchmarker
[beam_PerformanceTests_Spark] $ /bin/bash -xe /tmp/jenkins6466491190304263004.sh
+ rm -rf .env
[beam_PerformanceTests_Spark] $ /bin/bash -xe /tmp/jenkins1921546234116870363.sh
+ virtualenv .env --system-site-packages
New python executable in 

Installing setuptools, pip, wheel...done.
[beam_PerformanceTests_Spark] $ /bin/bash -xe /tmp/jenkins1350795406443771827.sh
+ git clone https://github.com/GoogleCloudPlatform/PerfKitBenchmarker.git
Cloning into 'PerfKitBenchmarker'...
[beam_PerformanceTests_Spark] $ /bin/bash -xe /tmp/jenkins1586794224047448646.sh
+ .env/bin/pip install -r PerfKitBenchmarker/requirements.txt
Requirement already satisfied: absl-py in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 14))
Requirement already satisfied: jinja2>=2.7 in 
/usr/local/lib/python2.7/dist-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 15))
Requirement already satisfied: setuptools in ./.env/lib/python2.7/site-packages 
(from -r PerfKitBenchmarker/requirements.txt (line 16))
Requirement already satisfied: colorlog[windows]==2.6.0 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 17))
Requirement already satisfied: blinker>=1.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 18))
Requirement already satisfied: futures>=3.0.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 19))
Requirement already satisfied: PyYAML==3.12 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 20))
Requirement already satisfied: pint>=0.7 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 21))
Requirement already satisfied: numpy in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 22))
Requirement already satisfied: functools32 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 23))
Requirement already satisfied: contextlib2>=0.5.1 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 24))
Requirement already satisfied: pywinrm in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 25))
Requirement already satisfied: six in 
/home/jenkins/.local/lib/python2.7/site-packages (from absl-py->-r 
PerfKitBenchmarker/requirements.txt (line 14))
Requirement already satisfied: MarkupSafe in 
/usr/local/lib/python2.7/dist-packages (from jinja2>=2.7->-r 
PerfKitBenchmarker/requirements.txt (line 15))
Requirement already satisfied: colorama; extra == "windows" in 
/usr/lib/python2.7/dist-packages (from colorlog[windows]==2.6.0->-r 
PerfKitBenchmarker/requirements.txt (line 17))
Requirement already satisfied: xmltodict in 

Build failed in Jenkins: beam_PerformanceTests_Python #839

[Apache Jenkins Server]

See 


Changes:

[zoy] Increasing BatchElements's max_batch_size to 10K

[lcwik] [BEAM-2273] Cleanup examples Maven Archetype to copy in a clean state

[XuMingmin] [BEAM-3526] KakfaIO support for finalizeCheckpoint() (#4481)

[kedin] Add Avro dependency to KafkaIO

--
Started by timer
[EnvInject] - Loading node environment variables.
Building remotely on beam7 (beam) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* 
 > +refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/*
 > git rev-parse origin/master^{commit} # timeout=10
Checking out Revision 71192743121d0b142828da870905369f15c64f15 (origin/master)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 71192743121d0b142828da870905369f15c64f15
Commit message: "Merge pull request #4503: Add Avro dependency to KafkaIO"
 > git rev-list d4a55d9f485a06a9fc0933884c36d38af3c69d3d # timeout=10
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
[EnvInject] - Executing scripts and injecting environment variables after the 
SCM step.
[EnvInject] - Injecting as environment variables the properties content 
SPARK_LOCAL_IP=127.0.0.1

[EnvInject] - Variables injected successfully.
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins6456047027741313524.sh
+ rm -rf PerfKitBenchmarker
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins1644363041463065565.sh
+ rm -rf .env
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins2048723133408611392.sh
+ virtualenv .env --system-site-packages
New python executable in 

Installing setuptools, pip, wheel...done.
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins4794801852816059528.sh
+ git clone https://github.com/GoogleCloudPlatform/PerfKitBenchmarker.git
Cloning into 'PerfKitBenchmarker'...
[beam_PerformanceTests_Python] $ /bin/bash -xe /tmp/jenkins855725714432159011.sh
+ .env/bin/pip install -r PerfKitBenchmarker/requirements.txt
Requirement already satisfied: absl-py in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 14))
Requirement already satisfied: jinja2>=2.7 in 
/usr/local/lib/python2.7/dist-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 15))
Requirement already satisfied: setuptools in ./.env/lib/python2.7/site-packages 
(from -r PerfKitBenchmarker/requirements.txt (line 16))
Requirement already satisfied: colorlog[windows]==2.6.0 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 17))
Requirement already satisfied: blinker>=1.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 18))
Requirement already satisfied: futures>=3.0.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 19))
Requirement already satisfied: PyYAML==3.12 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 20))
Requirement already satisfied: pint>=0.7 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 21))
Requirement already satisfied: numpy in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 22))
Requirement already satisfied: functools32 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 23))
Requirement already satisfied: contextlib2>=0.5.1 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 24))
Requirement already satisfied: pywinrm in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 25))
Requirement already satisfied: six in 
/home/jenkins/.local/lib/python2.7/site-packages (from absl-py->-r 
PerfKitBenchmarker/requirements.txt (line 14))
Requirement already satisfied: MarkupSafe>=0.23 in 
/usr/local/lib/python2.7/dist-packages (from jinja2>=2.7->-r 
PerfKitBenchmarker/requirements.txt (line 15))
Requirement already satisfied: colorama; extra == "windows" in 
/usr/lib/python2.7/dist-packages (from colorlog[windows]==2.6.0->-r 
PerfKitBenchmarker/requirements.txt (line 17))
Requirement already satisfied: xmltodict in 

Build failed in Jenkins: beam_PerformanceTests_TextIOIT #71

[Apache Jenkins Server]

See 


Changes:

[zoy] Increasing BatchElements's max_batch_size to 10K

[lcwik] [BEAM-2273] Cleanup examples Maven Archetype to copy in a clean state

[XuMingmin] [BEAM-3526] KakfaIO support for finalizeCheckpoint() (#4481)

[kedin] Add Avro dependency to KafkaIO

--
Started by timer
[EnvInject] - Loading node environment variables.
Building remotely on beam1 (beam) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* 
 > +refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/*
 > git rev-parse origin/master^{commit} # timeout=10
Checking out Revision 71192743121d0b142828da870905369f15c64f15 (origin/master)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 71192743121d0b142828da870905369f15c64f15
Commit message: "Merge pull request #4503: Add Avro dependency to KafkaIO"
 > git rev-list d4a55d9f485a06a9fc0933884c36d38af3c69d3d # timeout=10
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
[EnvInject] - Executing scripts and injecting environment variables after the 
SCM step.
[EnvInject] - Injecting as environment variables the properties content 
SPARK_LOCAL_IP=127.0.0.1

[EnvInject] - Variables injected successfully.
[beam_PerformanceTests_TextIOIT] $ /bin/bash -xe 
/tmp/jenkins4824376607788105297.sh
+ rm -rf PerfKitBenchmarker
[beam_PerformanceTests_TextIOIT] $ /bin/bash -xe 
/tmp/jenkins9188854482637368099.sh
+ rm -rf .env
[beam_PerformanceTests_TextIOIT] $ /bin/bash -xe 
/tmp/jenkins537228477192861770.sh
+ virtualenv .env --system-site-packages
New python executable in 

Installing setuptools, pip, wheel...done.
[beam_PerformanceTests_TextIOIT] $ /bin/bash -xe 
/tmp/jenkins3401119658047595565.sh
+ git clone https://github.com/GoogleCloudPlatform/PerfKitBenchmarker.git
Cloning into 'PerfKitBenchmarker'...
[beam_PerformanceTests_TextIOIT] $ /bin/bash -xe 
/tmp/jenkins5736694369862812388.sh
+ .env/bin/pip install -r PerfKitBenchmarker/requirements.txt
Requirement already satisfied: absl-py in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 14))
Requirement already satisfied: jinja2>=2.7 in 
/usr/local/lib/python2.7/dist-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 15))
Requirement already satisfied: setuptools in ./.env/lib/python2.7/site-packages 
(from -r PerfKitBenchmarker/requirements.txt (line 16))
Requirement already satisfied: colorlog[windows]==2.6.0 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 17))
Requirement already satisfied: blinker>=1.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 18))
Requirement already satisfied: futures>=3.0.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 19))
Requirement already satisfied: PyYAML==3.12 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 20))
Requirement already satisfied: pint>=0.7 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 21))
Requirement already satisfied: numpy in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 22))
Requirement already satisfied: functools32 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 23))
Requirement already satisfied: contextlib2>=0.5.1 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 24))
Requirement already satisfied: pywinrm in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 25))
Requirement already satisfied: six in 
/home/jenkins/.local/lib/python2.7/site-packages (from absl-py->-r 
PerfKitBenchmarker/requirements.txt (line 14))
Requirement already satisfied: MarkupSafe in 
/usr/local/lib/python2.7/dist-packages (from jinja2>=2.7->-r 
PerfKitBenchmarker/requirements.txt (line 15))
Requirement already satisfied: colorama; extra == "windows" in 
/usr/lib/python2.7/dist-packages (from colorlog[windows]==2.6.0->-r 
PerfKitBenchmarker/requirements.txt (line 17))
Requirement already satisfied: xmltodict in 

Build failed in Jenkins: beam_PerformanceTests_Compressed_TextIOIT #63

[Apache Jenkins Server]

See 


Changes:

[zoy] Increasing BatchElements's max_batch_size to 10K

[lcwik] [BEAM-2273] Cleanup examples Maven Archetype to copy in a clean state

[XuMingmin] [BEAM-3526] KakfaIO support for finalizeCheckpoint() (#4481)

[kedin] Add Avro dependency to KafkaIO

--
Started by timer
[EnvInject] - Loading node environment variables.
Building remotely on beam6 (beam) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* 
 > +refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/*
 > git rev-parse origin/master^{commit} # timeout=10
Checking out Revision 71192743121d0b142828da870905369f15c64f15 (origin/master)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 71192743121d0b142828da870905369f15c64f15
Commit message: "Merge pull request #4503: Add Avro dependency to KafkaIO"
 > git rev-list d4a55d9f485a06a9fc0933884c36d38af3c69d3d # timeout=10
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
[EnvInject] - Executing scripts and injecting environment variables after the 
SCM step.
[EnvInject] - Injecting as environment variables the properties content 
SPARK_LOCAL_IP=127.0.0.1

[EnvInject] - Variables injected successfully.
[beam_PerformanceTests_Compressed_TextIOIT] $ /bin/bash -xe 
/tmp/jenkins7298801133445650511.sh
+ rm -rf PerfKitBenchmarker
[beam_PerformanceTests_Compressed_TextIOIT] $ /bin/bash -xe 
/tmp/jenkins6444118591333607215.sh
+ rm -rf .env
[beam_PerformanceTests_Compressed_TextIOIT] $ /bin/bash -xe 
/tmp/jenkins5369157286590898522.sh
+ virtualenv .env --system-site-packages
New python executable in .env/bin/python
Installing setuptools, pip...done.
[beam_PerformanceTests_Compressed_TextIOIT] $ /bin/bash -xe 
/tmp/jenkins4309419115273450259.sh
+ git clone https://github.com/GoogleCloudPlatform/PerfKitBenchmarker.git
Cloning into 'PerfKitBenchmarker'...
[beam_PerformanceTests_Compressed_TextIOIT] $ /bin/bash -xe 
/tmp/jenkins2187161493073001920.sh
+ .env/bin/pip install -r PerfKitBenchmarker/requirements.txt
Requirement already satisfied (use --upgrade to upgrade): absl-py in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 14))
Requirement already satisfied (use --upgrade to upgrade): jinja2>=2.7 in 
/usr/local/lib/python2.7/dist-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 15))
Requirement already satisfied (use --upgrade to upgrade): setuptools in 
./.env/lib/python2.7/site-packages (from -r PerfKitBenchmarker/requirements.txt 
(line 16))
Requirement already satisfied (use --upgrade to upgrade): 
colorlog[windows]==2.6.0 in /home/jenkins/.local/lib/python2.7/site-packages 
(from -r PerfKitBenchmarker/requirements.txt (line 17))
  Installing extra requirements: 'windows'
Requirement already satisfied (use --upgrade to upgrade): blinker>=1.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 18))
Requirement already satisfied (use --upgrade to upgrade): futures>=3.0.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 19))
Requirement already satisfied (use --upgrade to upgrade): PyYAML==3.12 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 20))
Requirement already satisfied (use --upgrade to upgrade): pint>=0.7 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 21))
Requirement already satisfied (use --upgrade to upgrade): numpy in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 22))
Requirement already satisfied (use --upgrade to upgrade): functools32 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 23))
Requirement already satisfied (use --upgrade to upgrade): contextlib2>=0.5.1 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 24))
Requirement already satisfied (use --upgrade to upgrade): pywinrm in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 25))
Cleaning up...
[beam_PerformanceTests_Compressed_TextIOIT] $ /bin/bash -xe 
/tmp/jenkins3490935505498529585.sh
+ .env/bin/pip install -e 'src/sdks/python/[gcp,test]'

Build failed in Jenkins: beam_PerformanceTests_TFRecordIOIT #63

[Apache Jenkins Server]

See 


Changes:

[zoy] Increasing BatchElements's max_batch_size to 10K

[lcwik] [BEAM-2273] Cleanup examples Maven Archetype to copy in a clean state

[XuMingmin] [BEAM-3526] KakfaIO support for finalizeCheckpoint() (#4481)

[kedin] Add Avro dependency to KafkaIO

--
Started by timer
[EnvInject] - Loading node environment variables.
Building remotely on beam4 (beam) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* 
 > +refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/*
 > git rev-parse origin/master^{commit} # timeout=10
Checking out Revision 71192743121d0b142828da870905369f15c64f15 (origin/master)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 71192743121d0b142828da870905369f15c64f15
Commit message: "Merge pull request #4503: Add Avro dependency to KafkaIO"
 > git rev-list d4a55d9f485a06a9fc0933884c36d38af3c69d3d # timeout=10
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
[EnvInject] - Executing scripts and injecting environment variables after the 
SCM step.
[EnvInject] - Injecting as environment variables the properties content 
SPARK_LOCAL_IP=127.0.0.1

[EnvInject] - Variables injected successfully.
[beam_PerformanceTests_TFRecordIOIT] $ /bin/bash -xe 
/tmp/jenkins4041198055038685363.sh
+ rm -rf PerfKitBenchmarker
[beam_PerformanceTests_TFRecordIOIT] $ /bin/bash -xe 
/tmp/jenkins7866525248412368182.sh
+ rm -rf .env
[beam_PerformanceTests_TFRecordIOIT] $ /bin/bash -xe 
/tmp/jenkins2145219136281204943.sh
+ virtualenv .env --system-site-packages
New python executable in .env/bin/python
Installing setuptools, pip...done.
[beam_PerformanceTests_TFRecordIOIT] $ /bin/bash -xe 
/tmp/jenkins7576623018551659632.sh
+ git clone https://github.com/GoogleCloudPlatform/PerfKitBenchmarker.git
Cloning into 'PerfKitBenchmarker'...
[beam_PerformanceTests_TFRecordIOIT] $ /bin/bash -xe 
/tmp/jenkins1056739234013370747.sh
+ .env/bin/pip install -r PerfKitBenchmarker/requirements.txt
Requirement already satisfied (use --upgrade to upgrade): absl-py in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 14))
Requirement already satisfied (use --upgrade to upgrade): jinja2>=2.7 in 
/usr/local/lib/python2.7/dist-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 15))
Requirement already satisfied (use --upgrade to upgrade): setuptools in 
./.env/lib/python2.7/site-packages (from -r PerfKitBenchmarker/requirements.txt 
(line 16))
Requirement already satisfied (use --upgrade to upgrade): 
colorlog[windows]==2.6.0 in /home/jenkins/.local/lib/python2.7/site-packages 
(from -r PerfKitBenchmarker/requirements.txt (line 17))
  Installing extra requirements: 'windows'
Requirement already satisfied (use --upgrade to upgrade): blinker>=1.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 18))
Requirement already satisfied (use --upgrade to upgrade): futures>=3.0.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 19))
Requirement already satisfied (use --upgrade to upgrade): PyYAML==3.12 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 20))
Requirement already satisfied (use --upgrade to upgrade): pint>=0.7 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 21))
Requirement already satisfied (use --upgrade to upgrade): numpy in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 22))
Requirement already satisfied (use --upgrade to upgrade): functools32 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 23))
Requirement already satisfied (use --upgrade to upgrade): contextlib2>=0.5.1 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 24))
Requirement already satisfied (use --upgrade to upgrade): pywinrm in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 25))
Cleaning up...
[beam_PerformanceTests_TFRecordIOIT] $ /bin/bash -xe 
/tmp/jenkins4995329918938527087.sh
+ .env/bin/pip install -e 'src/sdks/python/[gcp,test]'
Obtaining 

Jenkins build is back to normal : beam_PostCommit_Java_MavenInstall #5770

[Apache Jenkins Server]

See 

[jira] [Updated] (BEAM-3547) [SQL] Nested Query Generates Incompatible Trigger

[Anton Kedin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3547?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Anton Kedin updated BEAM-3547:
--
Description: 
>From [https://stackoverflow.com/questions/48335383/nested-queries-in-beam-sql] 
>:

 

SQL:
{code:java}
PCollection Query_Output = Query.apply(
BeamSql.queryMulti("Select Orders.OrderID From Orders Where 
Orders.CustomerID IN (Select Customers.CustomerID From Customers WHERE 
Customers.CustomerID = 2)"));{code}
 

Error:
{code:java}
org.apache.beam.sdk.extensions.sql.impl.planner.BeamQueryPlanner 
validateAndConvert
INFO: SQL:
SELECT `Orders`.`OrderID`
FROM `Orders` AS `Orders`
WHERE `Orders`.`CustomerID` IN (SELECT `Customers`.`CustomerID`
FROM `Customers` AS `Customers`
WHERE `Customers`.`CustomerID` = 2)
Jan 19, 2018 11:56:36 AM 
org.apache.beam.sdk.extensions.sql.impl.planner.BeamQueryPlanner 
convertToBeamRel
INFO: SQLPlan>
LogicalProject(OrderID=[$0])
  LogicalJoin(condition=[=($1, $3)], joinType=[inner])
LogicalTableScan(table=[[Orders]])
LogicalAggregate(group=[{0}])
  LogicalProject(CustomerID=[$0])
LogicalFilter(condition=[=($0, 2)])
  LogicalTableScan(table=[[Customers]])

Exception in thread "main" java.lang.IllegalStateException: 
java.lang.IllegalStateException: Inputs to Flatten had incompatible triggers: 
DefaultTrigger, Repeatedly.forever(AfterWatermark.pastEndOfWindow())
at 
org.apache.beam.sdk.extensions.sql.BeamSql$QueryTransform.expand(BeamSql.java:165)
at 
org.apache.beam.sdk.extensions.sql.BeamSql$QueryTransform.expand(BeamSql.java:116)
at org.apache.beam.sdk.Pipeline.applyInternal(Pipeline.java:533)
at org.apache.beam.sdk.Pipeline.applyTransform(Pipeline.java:465)
at 
org.apache.beam.sdk.values.PCollectionTuple.apply(PCollectionTuple.java:160)
at com.bitwise.cloud.ExampleOfJoins.main(ExampleOfJoins.java:91)
Caused by: java.lang.IllegalStateException: Inputs to Flatten had incompatible 
triggers: DefaultTrigger, Repeatedly.forever(AfterWatermark.pastEndOfWindow())
at 
org.apache.beam.sdk.transforms.Flatten$PCollections.expand(Flatten.java:123)
at 
org.apache.beam.sdk.transforms.Flatten$PCollections.expand(Flatten.java:101)
at org.apache.beam.sdk.Pipeline.applyInternal(Pipeline.java:533)
at org.apache.beam.sdk.Pipeline.applyTransform(Pipeline.java:465)
at 
org.apache.beam.sdk.values.PCollectionList.apply(PCollectionList.java:182)
at 
org.apache.beam.sdk.transforms.join.CoGroupByKey.expand(CoGroupByKey.java:124)
at 
org.apache.beam.sdk.transforms.join.CoGroupByKey.expand(CoGroupByKey.java:74)
at org.apache.beam.sdk.Pipeline.applyInternal(Pipeline.java:533)
at org.apache.beam.sdk.Pipeline.applyTransform(Pipeline.java:465)
at 
org.apache.beam.sdk.transforms.join.KeyedPCollectionTuple.apply(KeyedPCollectionTuple.java:107)
at org.apache.beam.sdk.extensions.joinlibrary.Join.innerJoin(Join.java:59)
at 
org.apache.beam.sdk.extensions.sql.impl.rel.BeamJoinRel.standardJoin(BeamJoinRel.java:217)
at 
org.apache.beam.sdk.extensions.sql.impl.rel.BeamJoinRel.buildBeamPipeline(BeamJoinRel.java:161)
at 
org.apache.beam.sdk.extensions.sql.impl.rel.BeamProjectRel.buildBeamPipeline(BeamProjectRel.java:68)
at 
org.apache.beam.sdk.extensions.sql.BeamSql$QueryTransform.expand(BeamSql.java:163)
... 5 more{code}

  was:
>From 
>[https://beam.apache.org/documentation/programming-guide/#groupbykey-and-unbounded-pcollections]
> :

 

SQL:
{code:java}
PCollection Query_Output = Query.apply(
BeamSql.queryMulti("Select Orders.OrderID From Orders Where 
Orders.CustomerID IN (Select Customers.CustomerID From Customers WHERE 
Customers.CustomerID = 2)"));{code}
 

Error:
{code:java}
org.apache.beam.sdk.extensions.sql.impl.planner.BeamQueryPlanner 
validateAndConvert
INFO: SQL:
SELECT `Orders`.`OrderID`
FROM `Orders` AS `Orders`
WHERE `Orders`.`CustomerID` IN (SELECT `Customers`.`CustomerID`
FROM `Customers` AS `Customers`
WHERE `Customers`.`CustomerID` = 2)
Jan 19, 2018 11:56:36 AM 
org.apache.beam.sdk.extensions.sql.impl.planner.BeamQueryPlanner 
convertToBeamRel
INFO: SQLPlan>
LogicalProject(OrderID=[$0])
  LogicalJoin(condition=[=($1, $3)], joinType=[inner])
LogicalTableScan(table=[[Orders]])
LogicalAggregate(group=[{0}])
  LogicalProject(CustomerID=[$0])
LogicalFilter(condition=[=($0, 2)])
  LogicalTableScan(table=[[Customers]])

Exception in thread "main" java.lang.IllegalStateException: 
java.lang.IllegalStateException: Inputs to Flatten had incompatible triggers: 
DefaultTrigger, Repeatedly.forever(AfterWatermark.pastEndOfWindow())
at 
org.apache.beam.sdk.extensions.sql.BeamSql$QueryTransform.expand(BeamSql.java:165)
at 
org.apache.beam.sdk.extensions.sql.BeamSql$QueryTransform.expand(BeamSql.java:116)
at org.apache.beam.sdk.Pipeline.applyInternal(Pipeline.java:533)
at 

Jenkins build is unstable: beam_PostCommit_Java_ValidatesRunner_Spark #4003

[Apache Jenkins Server]

See 

[jira] [Commented] (BEAM-1866) Fn API support for Metrics

[Kenneth Knowles (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1866?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16341731#comment-16341731
 ] 

Kenneth Knowles commented on BEAM-1866:
---

Deleted a comment that was just a giant diff. Robert's PR was merged.

> Fn API support for Metrics
> --
>
> Key: BEAM-1866
> URL: https://issues.apache.org/jira/browse/BEAM-1866
> Project: Beam
>  Issue Type: New Feature
>  Components: beam-model
>Reporter: Daniel Halperin
>Assignee: Kenneth Knowles
>Priority: Major
>  Labels: portability
>
> As part of the Fn API work, we need to define a Metrics interface between the 
> Runner and the SDK. Right now, Metrics are simply lost.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (BEAM-1866) Fn API support for Metrics

[Kenneth Knowles (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1866?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16341733#comment-16341733
 ] 

Kenneth Knowles commented on BEAM-1866:
---

For reference in the Fn API design, here is the pre-portability design doc: 
https://docs.google.com/document/d/1voyUIQ2DrWkoY-BsJwM8YvF4gGKB76CDG8BYL8XBc7A/edit#

> Fn API support for Metrics
> --
>
> Key: BEAM-1866
> URL: https://issues.apache.org/jira/browse/BEAM-1866
> Project: Beam
>  Issue Type: New Feature
>  Components: beam-model
>Reporter: Daniel Halperin
>Assignee: Kenneth Knowles
>Priority: Major
>  Labels: portability
>
> As part of the Fn API work, we need to define a Metrics interface between the 
> Runner and the SDK. Right now, Metrics are simply lost.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Comment Edited] (BEAM-1866) Fn API support for Metrics

[Kenneth Knowles (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1866?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16341733#comment-16341733
 ] 

Kenneth Knowles edited comment on BEAM-1866 at 1/26/18 10:49 PM:
-

For reference in the Fn API design, here is the pre-portability design doc: 
https://s.apache.org/beam-metrics-api


was (Author: kenn):
For reference in the Fn API design, here is the pre-portability design doc: 
https://docs.google.com/document/d/1voyUIQ2DrWkoY-BsJwM8YvF4gGKB76CDG8BYL8XBc7A/edit#

> Fn API support for Metrics
> --
>
> Key: BEAM-1866
> URL: https://issues.apache.org/jira/browse/BEAM-1866
> Project: Beam
>  Issue Type: New Feature
>  Components: beam-model
>Reporter: Daniel Halperin
>Assignee: Kenneth Knowles
>Priority: Major
>  Labels: portability
>
> As part of the Fn API work, we need to define a Metrics interface between the 
> Runner and the SDK. Right now, Metrics are simply lost.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Issue Comment Deleted] (BEAM-1866) Fn API support for Metrics

[Kenneth Knowles (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-1866?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kenneth Knowles updated BEAM-1866:
--
Comment: was deleted

(was: robertwb closed pull request #4344: [BEAM-1866] Plumb user metrics 
through Fn API.
URL: https://github.com/apache/beam/pull/4344
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git a/model/fn-execution/src/main/proto/beam_fn_api.proto 
b/model/fn-execution/src/main/proto/beam_fn_api.proto
index ca23c619f33..36ed4242d6c 100644
--- a/model/fn-execution/src/main/proto/beam_fn_api.proto
+++ b/model/fn-execution/src/main/proto/beam_fn_api.proto
@@ -280,11 +280,44 @@ message Metrics {
 
   // User defined metrics
   message User {
-// TODO: Define it.
+
+// A key for identifying a metric at the most granular level.
+message MetricKey {
+  // The step, if any, this metric is associated with.
+  string step = 1;
+
+  // (Required): The namespace of this metric.
+  string namespace = 2;
+
+  // (Required): The name of this metric.
+  string name = 3;
+}
+
+// Data associated with a counter metric.
+message CounterData {
+  int64 value = 1;
+}
+
+// Data associated with a distribution metric.
+message DistributionData {
+  int64 count = 1;
+  int64 sum = 2;
+  int64 min = 3;
+  int64 max = 4;
+}
+
+// (Required) The identifier for this metric.
+MetricKey key = 1;
+
+// (Required) The data for this metric.
+oneof data {
+  CounterData counter_data = 1001;
+  DistributionData distribution_data = 1002;
+}
   }
 
   map ptransforms = 1;
-  map user = 2;
+  repeated User user = 2;
 }
 
 message ProcessBundleProgressResponse {
diff --git a/sdks/python/apache_beam/metrics/cells.py 
b/sdks/python/apache_beam/metrics/cells.py
index ba840f752b1..2b5a4e4094d 100644
--- a/sdks/python/apache_beam/metrics/cells.py
+++ b/sdks/python/apache_beam/metrics/cells.py
@@ -28,6 +28,7 @@
 
 from apache_beam.metrics.metricbase import Counter
 from apache_beam.metrics.metricbase import Distribution
+from apache_beam.portability.api import beam_fn_api_pb2
 
 __all__ = ['DistributionResult']
 
@@ -286,10 +287,18 @@ def combine(self, other):
 new_min,
 new_max)
 
-  @classmethod
-  def singleton(cls, value):
+  @staticmethod
+  def singleton(value):
 return DistributionData(value, 1, value, value)
 
+  def to_runner_api(self):
+return beam_fn_api_pb2.Metrics.User.DistributionData(
+count=self.count, sum=self.sum, min=self.min, max=self.max)
+
+  @staticmethod
+  def from_runner_api(proto):
+return DistributionData(proto.sum, proto.count, proto.min, proto.max)
+
 
 class MetricAggregator(object):
   """For internal use only; no backwards-compatibility guarantees.
diff --git a/sdks/python/apache_beam/metrics/execution.py 
b/sdks/python/apache_beam/metrics/execution.py
index 1704b98d46a..9e547a983de 100644
--- a/sdks/python/apache_beam/metrics/execution.py
+++ b/sdks/python/apache_beam/metrics/execution.py
@@ -34,6 +34,8 @@
 
 from apache_beam.metrics.cells import CounterCell
 from apache_beam.metrics.cells import DistributionCell
+from apache_beam.metrics.metricbase import MetricName
+from apache_beam.portability.api import beam_fn_api_pb2
 
 
 class MetricKey(object):
@@ -63,6 +65,14 @@ def __str__(self):
   def __hash__(self):
 return hash((self.step, self.metric))
 
+  def to_runner_api(self):
+return beam_fn_api_pb2.Metrics.User.MetricKey(
+step=self.step, namespace=self.metric.namespace, name=self.metric.name)
+
+  @staticmethod
+  def from_runner_api(proto):
+return MetricKey(proto.step, MetricName(proto.namespace, proto.name))
+
 
 class MetricResult(object):
   """Keeps track of the status of a metric within a single bundle.
@@ -192,6 +202,20 @@ def get_cumulative(self):
 """
 return self._get_updates()
 
+  def to_runner_api(self):
+return (
+[beam_fn_api_pb2.Metrics.User(
+key=beam_fn_api_pb2.Metrics.User.MetricKey(
+step=self.step_name, namespace=k.namespace, name=k.name),
+counter_data=beam_fn_api_pb2.Metrics.User.CounterData(
+value=v.get_cumulative()))
+ for k, v in self.counters.items()] +
+[beam_fn_api_pb2.Metrics.User(
+key=beam_fn_api_pb2.Metrics.User.MetricKey(
+step=self.step_name, namespace=k.namespace, name=k.name),
+distribution_data=v.get_cumulative().to_runner_api())
+ for k, v in self.distributions.items()])
+
 
 class ScopedMetricsContainer(object):
 
diff --git a/sdks/python/apache_beam/runners/portability/fn_api_runner.py 

[jira] [Commented] (BEAM-3547) [SQL] Nested Query Generates Incompatible Trigger

[Anton Kedin (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-3547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16341713#comment-16341713
 ] 

Anton Kedin commented on BEAM-3547:
---

Looking at planner output, Calcite generated a LogicalAggregate operation, 
which will override the input PCollection's trigger.

This behavior should be changed to inherit windowing strategy / triggers from 
the input unless explicitly specified.

The doc with details: 
[https://docs.google.com/document/d/1RmyV9e1Qab-axsLI1WWpw5oGAJDv0X7y9OSnPnrZWJk/edit]

 

> [SQL] Nested Query Generates Incompatible Trigger
> -
>
> Key: BEAM-3547
> URL: https://issues.apache.org/jira/browse/BEAM-3547
> Project: Beam
>  Issue Type: Bug
>  Components: dsl-sql
>Reporter: Anton Kedin
>Priority: Major
>
> From 
> [https://beam.apache.org/documentation/programming-guide/#groupbykey-and-unbounded-pcollections]
>  :
>  
> SQL:
> {code:java}
> PCollection Query_Output = Query.apply(
> BeamSql.queryMulti("Select Orders.OrderID From Orders Where 
> Orders.CustomerID IN (Select Customers.CustomerID From Customers WHERE 
> Customers.CustomerID = 2)"));{code}
>  
> Error:
> {code:java}
> org.apache.beam.sdk.extensions.sql.impl.planner.BeamQueryPlanner 
> validateAndConvert
> INFO: SQL:
> SELECT `Orders`.`OrderID`
> FROM `Orders` AS `Orders`
> WHERE `Orders`.`CustomerID` IN (SELECT `Customers`.`CustomerID`
> FROM `Customers` AS `Customers`
> WHERE `Customers`.`CustomerID` = 2)
> Jan 19, 2018 11:56:36 AM 
> org.apache.beam.sdk.extensions.sql.impl.planner.BeamQueryPlanner 
> convertToBeamRel
> INFO: SQLPlan>
> LogicalProject(OrderID=[$0])
>   LogicalJoin(condition=[=($1, $3)], joinType=[inner])
> LogicalTableScan(table=[[Orders]])
> LogicalAggregate(group=[{0}])
>   LogicalProject(CustomerID=[$0])
> LogicalFilter(condition=[=($0, 2)])
>   LogicalTableScan(table=[[Customers]])
> Exception in thread "main" java.lang.IllegalStateException: 
> java.lang.IllegalStateException: Inputs to Flatten had incompatible triggers: 
> DefaultTrigger, Repeatedly.forever(AfterWatermark.pastEndOfWindow())
> at 
> org.apache.beam.sdk.extensions.sql.BeamSql$QueryTransform.expand(BeamSql.java:165)
> at 
> org.apache.beam.sdk.extensions.sql.BeamSql$QueryTransform.expand(BeamSql.java:116)
> at org.apache.beam.sdk.Pipeline.applyInternal(Pipeline.java:533)
> at org.apache.beam.sdk.Pipeline.applyTransform(Pipeline.java:465)
> at 
> org.apache.beam.sdk.values.PCollectionTuple.apply(PCollectionTuple.java:160)
> at com.bitwise.cloud.ExampleOfJoins.main(ExampleOfJoins.java:91)
> Caused by: java.lang.IllegalStateException: Inputs to Flatten had 
> incompatible triggers: DefaultTrigger, 
> Repeatedly.forever(AfterWatermark.pastEndOfWindow())
> at 
> org.apache.beam.sdk.transforms.Flatten$PCollections.expand(Flatten.java:123)
> at 
> org.apache.beam.sdk.transforms.Flatten$PCollections.expand(Flatten.java:101)
> at org.apache.beam.sdk.Pipeline.applyInternal(Pipeline.java:533)
> at org.apache.beam.sdk.Pipeline.applyTransform(Pipeline.java:465)
> at 
> org.apache.beam.sdk.values.PCollectionList.apply(PCollectionList.java:182)
> at 
> org.apache.beam.sdk.transforms.join.CoGroupByKey.expand(CoGroupByKey.java:124)
> at 
> org.apache.beam.sdk.transforms.join.CoGroupByKey.expand(CoGroupByKey.java:74)
> at org.apache.beam.sdk.Pipeline.applyInternal(Pipeline.java:533)
> at org.apache.beam.sdk.Pipeline.applyTransform(Pipeline.java:465)
> at 
> org.apache.beam.sdk.transforms.join.KeyedPCollectionTuple.apply(KeyedPCollectionTuple.java:107)
> at org.apache.beam.sdk.extensions.joinlibrary.Join.innerJoin(Join.java:59)
> at 
> org.apache.beam.sdk.extensions.sql.impl.rel.BeamJoinRel.standardJoin(BeamJoinRel.java:217)
> at 
> org.apache.beam.sdk.extensions.sql.impl.rel.BeamJoinRel.buildBeamPipeline(BeamJoinRel.java:161)
> at 
> org.apache.beam.sdk.extensions.sql.impl.rel.BeamProjectRel.buildBeamPipeline(BeamProjectRel.java:68)
> at 
> org.apache.beam.sdk.extensions.sql.BeamSql$QueryTransform.expand(BeamSql.java:163)
> ... 5 more{code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (BEAM-3547) [SQL] Nested Query Generates Incompatible Trigger

[Anton Kedin (JIRA)]

Anton Kedin created BEAM-3547:
-

 Summary: [SQL] Nested Query Generates Incompatible Trigger
 Key: BEAM-3547
 URL: https://issues.apache.org/jira/browse/BEAM-3547
 Project: Beam
  Issue Type: Bug
  Components: dsl-sql
Reporter: Anton Kedin


>From 
>[https://beam.apache.org/documentation/programming-guide/#groupbykey-and-unbounded-pcollections]
> :

 

SQL:
{code:java}
PCollection Query_Output = Query.apply(
BeamSql.queryMulti("Select Orders.OrderID From Orders Where 
Orders.CustomerID IN (Select Customers.CustomerID From Customers WHERE 
Customers.CustomerID = 2)"));{code}
 

Error:
{code:java}
org.apache.beam.sdk.extensions.sql.impl.planner.BeamQueryPlanner 
validateAndConvert
INFO: SQL:
SELECT `Orders`.`OrderID`
FROM `Orders` AS `Orders`
WHERE `Orders`.`CustomerID` IN (SELECT `Customers`.`CustomerID`
FROM `Customers` AS `Customers`
WHERE `Customers`.`CustomerID` = 2)
Jan 19, 2018 11:56:36 AM 
org.apache.beam.sdk.extensions.sql.impl.planner.BeamQueryPlanner 
convertToBeamRel
INFO: SQLPlan>
LogicalProject(OrderID=[$0])
  LogicalJoin(condition=[=($1, $3)], joinType=[inner])
LogicalTableScan(table=[[Orders]])
LogicalAggregate(group=[{0}])
  LogicalProject(CustomerID=[$0])
LogicalFilter(condition=[=($0, 2)])
  LogicalTableScan(table=[[Customers]])

Exception in thread "main" java.lang.IllegalStateException: 
java.lang.IllegalStateException: Inputs to Flatten had incompatible triggers: 
DefaultTrigger, Repeatedly.forever(AfterWatermark.pastEndOfWindow())
at 
org.apache.beam.sdk.extensions.sql.BeamSql$QueryTransform.expand(BeamSql.java:165)
at 
org.apache.beam.sdk.extensions.sql.BeamSql$QueryTransform.expand(BeamSql.java:116)
at org.apache.beam.sdk.Pipeline.applyInternal(Pipeline.java:533)
at org.apache.beam.sdk.Pipeline.applyTransform(Pipeline.java:465)
at 
org.apache.beam.sdk.values.PCollectionTuple.apply(PCollectionTuple.java:160)
at com.bitwise.cloud.ExampleOfJoins.main(ExampleOfJoins.java:91)
Caused by: java.lang.IllegalStateException: Inputs to Flatten had incompatible 
triggers: DefaultTrigger, Repeatedly.forever(AfterWatermark.pastEndOfWindow())
at 
org.apache.beam.sdk.transforms.Flatten$PCollections.expand(Flatten.java:123)
at 
org.apache.beam.sdk.transforms.Flatten$PCollections.expand(Flatten.java:101)
at org.apache.beam.sdk.Pipeline.applyInternal(Pipeline.java:533)
at org.apache.beam.sdk.Pipeline.applyTransform(Pipeline.java:465)
at 
org.apache.beam.sdk.values.PCollectionList.apply(PCollectionList.java:182)
at 
org.apache.beam.sdk.transforms.join.CoGroupByKey.expand(CoGroupByKey.java:124)
at 
org.apache.beam.sdk.transforms.join.CoGroupByKey.expand(CoGroupByKey.java:74)
at org.apache.beam.sdk.Pipeline.applyInternal(Pipeline.java:533)
at org.apache.beam.sdk.Pipeline.applyTransform(Pipeline.java:465)
at 
org.apache.beam.sdk.transforms.join.KeyedPCollectionTuple.apply(KeyedPCollectionTuple.java:107)
at org.apache.beam.sdk.extensions.joinlibrary.Join.innerJoin(Join.java:59)
at 
org.apache.beam.sdk.extensions.sql.impl.rel.BeamJoinRel.standardJoin(BeamJoinRel.java:217)
at 
org.apache.beam.sdk.extensions.sql.impl.rel.BeamJoinRel.buildBeamPipeline(BeamJoinRel.java:161)
at 
org.apache.beam.sdk.extensions.sql.impl.rel.BeamProjectRel.buildBeamPipeline(BeamProjectRel.java:68)
at 
org.apache.beam.sdk.extensions.sql.BeamSql$QueryTransform.expand(BeamSql.java:163)
... 5 more{code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (BEAM-3546) Fn API metrics in Dataflow

[Kenneth Knowles (JIRA)]

Kenneth Knowles created BEAM-3546:
-

 Summary: Fn API metrics in Dataflow
 Key: BEAM-3546
 URL: https://issues.apache.org/jira/browse/BEAM-3546
 Project: Beam
  Issue Type: Sub-task
  Components: runner-dataflow
Reporter: Kenneth Knowles
Assignee: Kenneth Knowles






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (BEAM-3545) Fn API metrics in Go SDK harness

[Kenneth Knowles (JIRA)]

Kenneth Knowles created BEAM-3545:
-

 Summary: Fn API metrics in Go SDK harness
 Key: BEAM-3545
 URL: https://issues.apache.org/jira/browse/BEAM-3545
 Project: Beam
  Issue Type: Sub-task
  Components: sdk-java-harness
Reporter: Kenneth Knowles
Assignee: Henning Rohde






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (BEAM-3543) Fn API metrics in Java SDK Harness

[Kenneth Knowles (JIRA)]

Kenneth Knowles created BEAM-3543:
-

 Summary: Fn API metrics in Java SDK Harness
 Key: BEAM-3543
 URL: https://issues.apache.org/jira/browse/BEAM-3543
 Project: Beam
  Issue Type: Sub-task
  Components: sdk-java-harness
Reporter: Kenneth Knowles
Assignee: Kenneth Knowles






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (BEAM-3544) Fn API metrics in Python SDK Harness

[Kenneth Knowles (JIRA)]

Kenneth Knowles created BEAM-3544:
-

 Summary: Fn API metrics in Python SDK Harness
 Key: BEAM-3544
 URL: https://issues.apache.org/jira/browse/BEAM-3544
 Project: Beam
  Issue Type: Sub-task
  Components: sdk-java-harness
Reporter: Kenneth Knowles
Assignee: Robert Bradshaw






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Comment Edited] (BEAM-3310) Push metrics to a backend in an runner agnostic way

[Luke Cwik (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-3310?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16341696#comment-16341696
 ] 

Luke Cwik edited comment on BEAM-3310 at 1/26/18 10:23 PM:
---

[~echauchot], from the design doc or this Jira which which approach you were 
targeting?


was (Author: lcwik):
[~echauchot], from the design doc or this Jira which which approach you were 
targeting.

> Push metrics to a backend in an runner agnostic way
> ---
>
> Key: BEAM-3310
> URL: https://issues.apache.org/jira/browse/BEAM-3310
> Project: Beam
>  Issue Type: New Feature
>  Components: sdk-java-core
>Reporter: Etienne Chauchot
>Assignee: Etienne Chauchot
>Priority: Major
>
> The idea is to avoid relying on the runners to provide access to the metrics 
> (either at the end of the pipeline or while it runs) because they don't have 
> all the same capabilities towards metrics (e.g. spark runner configures sinks 
>  like csv, graphite or in memory sinks using the spark engine conf). The 
> target is to push the metrics in the common runner code so that no matter the 
> chosen runner, a user can get his metrics out of beam.
> Here is the link to the discussion thread on the dev ML: 
> https://lists.apache.org/thread.html/01a80d62f2df6b84bfa41f05e15fda900178f882877c294fed8be91e@%3Cdev.beam.apache.org%3E
> And the design doc:
> https://s.apache.org/runner_independent_metrics_extraction



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (BEAM-1866) Fn API support for Metrics

[Kenneth Knowles (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-1866?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kenneth Knowles reassigned BEAM-1866:
-

Assignee: Kenneth Knowles

> Fn API support for Metrics
> --
>
> Key: BEAM-1866
> URL: https://issues.apache.org/jira/browse/BEAM-1866
> Project: Beam
>  Issue Type: New Feature
>  Components: beam-model
>Reporter: Daniel Halperin
>Assignee: Kenneth Knowles
>Priority: Major
>  Labels: portability
>
> As part of the Fn API work, we need to define a Metrics interface between the 
> Runner and the SDK. Right now, Metrics are simply lost.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (BEAM-3310) Push metrics to a backend in an runner agnostic way

[Luke Cwik (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-3310?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16341696#comment-16341696
 ] 

Luke Cwik commented on BEAM-3310:
-

[~echauchot], from the design doc or this Jira which which approach you were 
targeting.

> Push metrics to a backend in an runner agnostic way
> ---
>
> Key: BEAM-3310
> URL: https://issues.apache.org/jira/browse/BEAM-3310
> Project: Beam
>  Issue Type: New Feature
>  Components: sdk-java-core
>Reporter: Etienne Chauchot
>Assignee: Etienne Chauchot
>Priority: Major
>
> The idea is to avoid relying on the runners to provide access to the metrics 
> (either at the end of the pipeline or while it runs) because they don't have 
> all the same capabilities towards metrics (e.g. spark runner configures sinks 
>  like csv, graphite or in memory sinks using the spark engine conf). The 
> target is to push the metrics in the common runner code so that no matter the 
> chosen runner, a user can get his metrics out of beam.
> Here is the link to the discussion thread on the dev ML: 
> https://lists.apache.org/thread.html/01a80d62f2df6b84bfa41f05e15fda900178f882877c294fed8be91e@%3Cdev.beam.apache.org%3E
> And the design doc:
> https://s.apache.org/runner_independent_metrics_extraction



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[beam] branch master updated (1274938 -> 7119274)

[kenn]

This is an automated email from the ASF dual-hosted git repository.

kenn pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from 1274938  [BEAM-3526] KakfaIO support for finalizeCheckpoint() (#4481)
 add f210060  Add Avro dependency to KafkaIO
 new 7119274  Merge pull request #4503: Add Avro dependency to KafkaIO

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 sdks/java/io/kafka/pom.xml | 6 ++
 1 file changed, 6 insertions(+)

-- 
To stop receiving notification emails like this one, please contact
k...@apache.org.

[beam] 01/01: Merge pull request #4503: Add Avro dependency to KafkaIO

[kenn]

This is an automated email from the ASF dual-hosted git repository.

kenn pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git

commit 71192743121d0b142828da870905369f15c64f15
Merge: 1274938 f210060
Author: Kenn Knowles 
AuthorDate: Fri Jan 26 14:21:14 2018 -0800

Merge pull request #4503: Add Avro dependency to KafkaIO

 sdks/java/io/kafka/pom.xml | 6 ++
 1 file changed, 6 insertions(+)

-- 
To stop receiving notification emails like this one, please contact
k...@apache.org.

[jira] [Commented] (BEAM-3310) Push metrics to a backend in an runner agnostic way

[Luke Cwik (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-3310?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16341683#comment-16341683
 ] 

Luke Cwik commented on BEAM-3310:
-

 

This could work as an extension to a pipeline if metrics sinks were defined 
part of pipeline construction and then metrics events would be sent to SDKs 
either on the control stream or over the data plane.

 

Alternatively this could be baked inside an SDK harness but would be limited to 
only reporting metrics for that given SDK instead of across all SDKs.

> Push metrics to a backend in an runner agnostic way
> ---
>
> Key: BEAM-3310
> URL: https://issues.apache.org/jira/browse/BEAM-3310
> Project: Beam
>  Issue Type: New Feature
>  Components: sdk-java-core
>Reporter: Etienne Chauchot
>Assignee: Etienne Chauchot
>Priority: Major
>
> The idea is to avoid relying on the runners to provide access to the metrics 
> (either at the end of the pipeline or while it runs) because they don't have 
> all the same capabilities towards metrics (e.g. spark runner configures sinks 
>  like csv, graphite or in memory sinks using the spark engine conf). The 
> target is to push the metrics in the common runner code so that no matter the 
> chosen runner, a user can get his metrics out of beam.
> Here is the link to the discussion thread on the dev ML: 
> https://lists.apache.org/thread.html/01a80d62f2df6b84bfa41f05e15fda900178f882877c294fed8be91e@%3Cdev.beam.apache.org%3E
> And the design doc:
> https://s.apache.org/runner_independent_metrics_extraction



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (BEAM-1169) MetricsTest matchers should loosen expectations on attempted values

[Kenneth Knowles (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-1169?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kenneth Knowles updated BEAM-1169:
--
Summary: MetricsTest matchers should loosen expectations on attempted 
values  (was: MetricsTest matchers should loosen expectations on physical 
values)

> MetricsTest matchers should loosen expectations on attempted values
> ---
>
> Key: BEAM-1169
> URL: https://issues.apache.org/jira/browse/BEAM-1169
> Project: Beam
>  Issue Type: Sub-task
>  Components: beam-model, sdk-java-core, sdk-py-core
>Reporter: Ben Chambers
>Priority: Major
>
> We could use `atLeast(N)` rather than `equals(N)` for the attempted values, 
> but even that may be false without violating the behavior.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (BEAM-1618) Add Gauge metric type to Python SDK

[Kenneth Knowles (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1618?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16341675#comment-16341675
 ] 

Kenneth Knowles commented on BEAM-1618:
---

Noting that this should be implemented via Fn API metrics so that Java work 
will also need to go into the SDK harness.

> Add Gauge metric type to Python SDK
> ---
>
> Key: BEAM-1618
> URL: https://issues.apache.org/jira/browse/BEAM-1618
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-py-core
>Reporter: Aviem Zur
>Priority: Major
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (BEAM-2996) Metric names should not be null or empty

[Kenneth Knowles (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2996?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kenneth Knowles resolved BEAM-2996.
---
   Resolution: Fixed
Fix Version/s: 2.3.0

Looks like you fixed this.

> Metric names should not be null or empty
> 
>
> Key: BEAM-2996
> URL: https://issues.apache.org/jira/browse/BEAM-2996
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core, sdk-py-core
>Reporter: Ben Chambers
>Assignee: Ben Chambers
>Priority: Minor
> Fix For: 2.3.0
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (BEAM-1866) Fn API support for Metrics

[Kenneth Knowles (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-1866?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kenneth Knowles updated BEAM-1866:
--
Summary: Fn API support for Metrics  (was: FnAPI support for Metrics)

> Fn API support for Metrics
> --
>
> Key: BEAM-1866
> URL: https://issues.apache.org/jira/browse/BEAM-1866
> Project: Beam
>  Issue Type: New Feature
>  Components: beam-model
>Reporter: Daniel Halperin
>Priority: Major
>  Labels: portability
>
> As part of the Fn API work, we need to define a Metrics interface between the 
> Runner and the SDK. Right now, Metrics are simply lost.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (BEAM-3383) Create validates runner metrics tests

[Kenneth Knowles (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3383?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kenneth Knowles resolved BEAM-3383.
---
   Resolution: Fixed
Fix Version/s: 2.0.0

> Create validates runner metrics tests
> -
>
> Key: BEAM-3383
> URL: https://issues.apache.org/jira/browse/BEAM-3383
> Project: Beam
>  Issue Type: Test
>  Components: runner-core
>Reporter: Etienne Chauchot
>Assignee: Etienne Chauchot
>Priority: Major
> Fix For: 2.0.0
>
>
> We need to create ValidatesRunner tests for metrics support among runners. 
> Need to have at least tests in batch and streaming modes and 
> committed/attempted metrics + exclusions in case of no support.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (BEAM-3383) Create validates runner metrics tests

[Kenneth Knowles (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-3383?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16341634#comment-16341634
 ] 

Kenneth Knowles commented on BEAM-3383:
---

[https://github.com/apache/beam/blob/master/sdks/java/core/src/test/java/org/apache/beam/sdk/metrics/MetricsTest.java]
 seems to have many ValidatesRunner tests so I think this got resolved 
independently.

> Create validates runner metrics tests
> -
>
> Key: BEAM-3383
> URL: https://issues.apache.org/jira/browse/BEAM-3383
> Project: Beam
>  Issue Type: Test
>  Components: runner-core
>Reporter: Etienne Chauchot
>Assignee: Etienne Chauchot
>Priority: Major
>
> We need to create ValidatesRunner tests for metrics support among runners. 
> Need to have at least tests in batch and streaming modes and 
> committed/attempted metrics + exclusions in case of no support.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (BEAM-3310) Push metrics to a backend in an runner agnostic way

[Kenneth Knowles (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-3310?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16341632#comment-16341632
 ] 

Kenneth Knowles commented on BEAM-3310:
---

I am rather new to the design, but from a quick scan this would seem to be more 
in line with the way that the Artifact API is a separate API that can either be 
provided by the runner or can be stood up alongside.

> Push metrics to a backend in an runner agnostic way
> ---
>
> Key: BEAM-3310
> URL: https://issues.apache.org/jira/browse/BEAM-3310
> Project: Beam
>  Issue Type: New Feature
>  Components: sdk-java-core
>Reporter: Etienne Chauchot
>Assignee: Etienne Chauchot
>Priority: Major
>
> The idea is to avoid relying on the runners to provide access to the metrics 
> (either at the end of the pipeline or while it runs) because they don't have 
> all the same capabilities towards metrics (e.g. spark runner configures sinks 
>  like csv, graphite or in memory sinks using the spark engine conf). The 
> target is to push the metrics in the common runner code so that no matter the 
> chosen runner, a user can get his metrics out of beam.
> Here is the link to the discussion thread on the dev ML: 
> https://lists.apache.org/thread.html/01a80d62f2df6b84bfa41f05e15fda900178f882877c294fed8be91e@%3Cdev.beam.apache.org%3E
> And the design doc:
> https://s.apache.org/runner_independent_metrics_extraction



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (BEAM-3310) Push metrics to a backend in an runner agnostic way

[Kenneth Knowles (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-3310?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16341630#comment-16341630
 ] 

Kenneth Knowles commented on BEAM-3310:
---

[~robertwb] [~lcwik] I think this idea might conflict with the practice of 
pushing metrics with progress updates.

> Push metrics to a backend in an runner agnostic way
> ---
>
> Key: BEAM-3310
> URL: https://issues.apache.org/jira/browse/BEAM-3310
> Project: Beam
>  Issue Type: New Feature
>  Components: sdk-java-core
>Reporter: Etienne Chauchot
>Assignee: Etienne Chauchot
>Priority: Major
>
> The idea is to avoid relying on the runners to provide access to the metrics 
> (either at the end of the pipeline or while it runs) because they don't have 
> all the same capabilities towards metrics (e.g. spark runner configures sinks 
>  like csv, graphite or in memory sinks using the spark engine conf). The 
> target is to push the metrics in the common runner code so that no matter the 
> chosen runner, a user can get his metrics out of beam.
> Here is the link to the discussion thread on the dev ML: 
> https://lists.apache.org/thread.html/01a80d62f2df6b84bfa41f05e15fda900178f882877c294fed8be91e@%3Cdev.beam.apache.org%3E
> And the design doc:
> https://s.apache.org/runner_independent_metrics_extraction



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Jenkins build is back to normal : beam_PostRelease_NightlySnapshot #6

[Apache Jenkins Server]

See 

[jira] [Commented] (BEAM-2271) Release guide or pom.xml needs update to avoid releasing Python binary artifacts

[JIRA]

[ 
https://issues.apache.org/jira/browse/BEAM-2271?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16341596#comment-16341596
 ] 

Jean-Baptiste Onofré commented on BEAM-2271:


I created a PR about that. I think we are pretty close ;)

> Release guide or pom.xml needs update to avoid releasing Python binary 
> artifacts
> 
>
> Key: BEAM-2271
> URL: https://issues.apache.org/jira/browse/BEAM-2271
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py-core
>Reporter: Daniel Halperin
>Assignee: Jean-Baptiste Onofré
>Priority: Major
> Fix For: 2.3.0
>
>
> The following directories (and children) were discovered in 2.0.0-RC2 and 
> were present in 0.6.0.
> {code}
> sdks/python: build   dist.eggs   nose-1.3.7-py2.7.egg  (and child 
> contents)
> {code}
> Ideally, these artifacts, which are created during setup and testing, would 
> get created in the {{sdks/python/target/}} subfolder where they will 
> automatically get ignored. More info below.
> For 2.0.0, we will manually remove these files from the source release RC3+. 
> This should be fixed before the next release.
> Here is a list of other paths that get excluded, should they be useful.
> {code}
> 
> 
> 
> %regex[(?!((?!${project.build.directory}/)[^/]+/)*src/).*${project.build.directory}.*]
> 
> 
>  
> 
> 
> %regex[(?!((?!${project.build.directory}/)[^/]+/)*src/)(.*/)?maven-eclipse\.xml]
> 
> %regex[(?!((?!${project.build.directory}/)[^/]+/)*src/)(.*/)?\.project]
> 
> %regex[(?!((?!${project.build.directory}/)[^/]+/)*src/)(.*/)?\.classpath]
> 
> %regex[(?!((?!${project.build.directory}/)[^/]+/)*src/)(.*/)?[^/]*\.iws]
> 
> %regex[(?!((?!${project.build.directory}/)[^/]+/)*src/)(.*/)?\.idea(/.*)?]
> 
> %regex[(?!((?!${project.build.directory}/)[^/]+/)*src/)(.*/)?out(/.*)?]
> 
> %regex[(?!((?!${project.build.directory}/)[^/]+/)*src/)(.*/)?[^/]*\.ipr]
> 
> %regex[(?!((?!${project.build.directory}/)[^/]+/)*src/)(.*/)?[^/]*\.iml]
> 
> %regex[(?!((?!${project.build.directory}/)[^/]+/)*src/)(.*/)?\.settings(/.*)?]
> 
> %regex[(?!((?!${project.build.directory}/)[^/]+/)*src/)(.*/)?\.externalToolBuilders(/.*)?]
> 
> %regex[(?!((?!${project.build.directory}/)[^/]+/)*src/)(.*/)?\.deployables(/.*)?]
> 
> %regex[(?!((?!${project.build.directory}/)[^/]+/)*src/)(.*/)?\.wtpmodules(/.*)?]
> 
> 
> 
> %regex[(?!((?!${project.build.directory}/)[^/]+/)*src/)(.*/)?cobertura\.ser]
> 
> 
> 
> %regex[(?!((?!${project.build.directory}/)[^/]+/)*src/)(.*/)?pom\.xml\.releaseBackup]
> 
> %regex[(?!((?!${project.build.directory}/)[^/]+/)*src/)(.*/)?release\.properties]
>   
> {code}
> This list is stored inside of this jar, which you can find by tracking 
> maven-assembly-plugin from the root apache pom: 
> https://mvnrepository.com/artifact/org.apache.apache.resources/apache-source-release-assembly-descriptor/1.0.6
> http://svn.apache.org/repos/asf/maven/pom/tags/apache-18/pom.xml



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (BEAM-2273) mvn clean doesn't fully clean up archetypes.

[JIRA]

 [ 
https://issues.apache.org/jira/browse/BEAM-2273?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jean-Baptiste Onofré resolved BEAM-2273.

   Resolution: Fixed
Fix Version/s: 2.3.0

> mvn clean doesn't fully clean up archetypes.
> 
>
> Key: BEAM-2273
> URL: https://issues.apache.org/jira/browse/BEAM-2273
> Project: Beam
>  Issue Type: Bug
>  Components: build-system
>Reporter: Jason Kuster
>Assignee: Jean-Baptiste Onofré
>Priority: Major
> Fix For: 2.3.0
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (BEAM-2273) mvn clean doesn't fully clean up archetypes.

[JIRA]

[ 
https://issues.apache.org/jira/browse/BEAM-2273?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16341595#comment-16341595
 ] 

Jean-Baptiste Onofré commented on BEAM-2273:


Yes, I confirm that the starter archetype doesn't generate or copy code (the 
example archetype uses a script to copy the code from examples).

> mvn clean doesn't fully clean up archetypes.
> 
>
> Key: BEAM-2273
> URL: https://issues.apache.org/jira/browse/BEAM-2273
> Project: Beam
>  Issue Type: Bug
>  Components: build-system
>Reporter: Jason Kuster
>Assignee: Jean-Baptiste Onofré
>Priority: Major
> Fix For: 2.3.0
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (BEAM-2273) mvn clean doesn't fully clean up archetypes.

[JIRA]

[ 
https://issues.apache.org/jira/browse/BEAM-2273?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16341585#comment-16341585
 ] 

Jean-Baptiste Onofré commented on BEAM-2273:


AFAIR, the starter archetype doesn't generate any code. I will double check 
anyway.

> mvn clean doesn't fully clean up archetypes.
> 
>
> Key: BEAM-2273
> URL: https://issues.apache.org/jira/browse/BEAM-2273
> Project: Beam
>  Issue Type: Bug
>  Components: build-system
>Reporter: Jason Kuster
>Assignee: Jean-Baptiste Onofré
>Priority: Major
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (BEAM-3542) SamzaPipelineOptions probably shouldn't need maxSourceParallelism

[Kenneth Knowles (JIRA)]

Kenneth Knowles created BEAM-3542:
-

 Summary: SamzaPipelineOptions probably shouldn't need 
maxSourceParallelism
 Key: BEAM-3542
 URL: https://issues.apache.org/jira/browse/BEAM-3542
 Project: Beam
  Issue Type: Bug
  Components: runner-samza
Reporter: Kenneth Knowles


Let's continue to examine and make sure the runner is using unbounded sources 
in a Beam-ish consistent way. If it is necessary, that is OK too, but it seemed 
there might be things to clarify since the code review.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (BEAM-3541) Findbugs fixes for Samza runner

[Kenneth Knowles (JIRA)]

Kenneth Knowles created BEAM-3541:
-

 Summary: Findbugs fixes for Samza runner
 Key: BEAM-3541
 URL: https://issues.apache.org/jira/browse/BEAM-3541
 Project: Beam
  Issue Type: Bug
  Components: runner-samza
Reporter: Kenneth Knowles


This can be an umbrella task for specific bugs, or you could just go and clear 
them up. I recommend doing the Gradle build first since that will be the 
authoritative build ASAP.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (BEAM-3540) Gradle build for Samza runner

[Kenneth Knowles (JIRA)]

Kenneth Knowles created BEAM-3540:
-

 Summary: Gradle build for Samza runner
 Key: BEAM-3540
 URL: https://issues.apache.org/jira/browse/BEAM-3540
 Project: Beam
  Issue Type: Bug
  Components: runner-samza
Reporter: Kenneth Knowles






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[beam] 01/01: Merge pull request #4340: [BEAM-3079] add Samza runner

[kenn]

This is an automated email from the ASF dual-hosted git repository.

kenn pushed a commit to branch samza-runner
in repository https://gitbox.apache.org/repos/asf/beam.git

commit 1840a582dae00ad68e7021944350baabc86649be
Merge: 0f554fd f5c8299
Author: Kenn Knowles 
AuthorDate: Fri Jan 26 12:40:27 2018 -0800

Merge pull request #4340: [BEAM-3079] add Samza runner

 runners/pom.xml|   1 +
 runners/samza/.gitignore   |   1 +
 runners/samza/pom.xml  | 322 +
 .../beam/runners/samza/SamzaExecutionContext.java  |  36 ++
 .../beam/runners/samza/SamzaPipelineOptions.java   |  50 ++
 .../samza/SamzaPipelineOptionsValidator.java   |  37 ++
 .../beam/runners/samza/SamzaPipelineResult.java| 158 +++
 .../org/apache/beam/runners/samza/SamzaRunner.java | 170 +++
 .../beam/runners/samza/SamzaRunnerRegistrar.java   |  59 +++
 .../apache/beam/runners/samza/TestSamzaRunner.java |  76 +++
 .../runners/samza/adapter/BoundedSourceSystem.java | 470 +++
 .../samza/adapter/UnboundedSourceSystem.java   | 503 
 .../beam/runners/samza/adapter/package-info.java   |  22 +
 .../samza/metrics/DoFnRunnerWithMetrics.java   |  80 
 .../samza/metrics/FnWithMetricsWrapper.java|  53 +++
 .../samza/metrics/SamzaMetricsContainer.java   | 114 +
 .../beam/runners/samza/metrics/package-info.java   |  22 +
 .../apache/beam/runners/samza/package-info.java|  22 +
 .../apache/beam/runners/samza/runtime/DoFnOp.java  | 322 +
 .../beam/runners/samza/runtime/GroupByKeyOp.java   | 167 +++
 .../beam/runners/samza/runtime/KeyedTimerData.java | 104 +
 .../runners/samza/runtime/KvToKeyedWorkItemOp.java |  41 ++
 .../org/apache/beam/runners/samza/runtime/Op.java  |  64 +++
 .../beam/runners/samza/runtime/OpAdapter.java  | 144 ++
 .../beam/runners/samza/runtime/OpEmitter.java  |  33 ++
 .../beam/runners/samza/runtime/OpMessage.java  | 145 ++
 .../samza/runtime/OutputManagerFactory.java|  29 ++
 .../runners/samza/runtime/SamzaAssignContext.java  |  55 +++
 .../samza/runtime/SamzaStoreStateInternals.java| 518 +
 .../samza/runtime/SamzaTimerInternalsFactory.java  | 298 
 .../samza/runtime/SingletonKeyedWorkItem.java  |  52 +++
 .../beam/runners/samza/runtime/WindowAssignOp.java |  53 +++
 .../beam/runners/samza/runtime/package-info.java   |  22 +
 .../runners/samza/translation/ConfigBuilder.java   | 167 +++
 .../translation/FlattenPCollectionsTranslator.java |  89 
 .../samza/translation/GroupByKeyTranslator.java| 143 ++
 .../runners/samza/translation/PViewToIdMapper.java |  66 +++
 .../translation/ParDoBoundMultiTranslator.java | 179 +++
 .../samza/translation/ReadBoundedTranslator.java   |  37 ++
 .../samza/translation/ReadUnboundedTranslator.java |  37 ++
 .../samza/translation/SamzaPipelineTranslator.java | 106 +
 .../samza/translation/SamzaPublishView.java|  55 +++
 .../SamzaPublishViewTransformOverride.java | 113 +
 .../translation/SamzaPublishViewTranslator.java|  49 ++
 .../samza/translation/SamzaTransformOverrides.java |  39 ++
 .../samza/translation/TransformTranslator.java |  31 ++
 .../samza/translation/TranslationContext.java  | 177 +++
 .../samza/translation/WindowAssignTranslator.java  |  53 +++
 .../runners/samza/translation/package-info.java|  22 +
 .../beam/runners/samza/util/Base64Serializer.java  |  68 +++
 .../runners/samza/util/PipelineDotRenderer.java| 137 ++
 .../beam/runners/samza/util/package-info.java  |  22 +
 runners/samza/src/main/resources/log4j.properties  |  22 +
 .../samza/src/main/resources/samza-conf.properties |  42 ++
 .../samza/adapter/BoundedSourceSystemTest.java | 319 +
 .../runners/samza/adapter/TestBoundedSource.java   | 200 
 .../runners/samza/adapter/TestCheckpointMark.java  |  43 ++
 .../runners/samza/adapter/TestSourceHelpers.java   | 168 +++
 .../runners/samza/adapter/TestUnboundedSource.java | 213 +
 .../samza/adapter/UnboundedSourceSystemTest.java   | 384 +++
 .../samza/src/test/resources/log4j-test.properties |  26 ++
 61 files changed, 7250 insertions(+)

-- 
To stop receiving notification emails like this one, please contact
k...@apache.org.

[jira] [Comment Edited] (BEAM-3392) Verify default container image names in Python SDK

[Valentyn Tymofieiev (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-3392?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16341543#comment-16341543
 ] 

Valentyn Tymofieiev edited comment on BEAM-3392 at 1/26/18 8:31 PM:


Yes, it's an action to perform on a branch. The intention for the Jira was to 
do a one-time check for 2.3.0, and setting fix version to 2.3.0 meant that 
someone will look into this.

I agree that a release verification spreadsheet is a cleaner way to track this. 
Feel free to add an item to the spreadsheet and remove this issue from 2.3.0 
blockers. Also, we could add information about  the spreadsheet to our release 
documentation 
[https://beam.apache.org/contribute/release-guide/#vote-on-the-release-candidate].


was (Author: tvalentyn):
Yes, it's an action to perform on a branch. The intention for the Jira was to 
do a one-time check for 2.3.0, and setting fix version to 2.3.0 meant that 
someone will look into this.

I agree that a release verification spreadsheet is a cleaner way to track this. 
Feel free to add an item to the spreadsheet and remove it from 2.3.0 blockers. 
Also, we could add information about  the spreadsheet to our release 
documentation 
https://beam.apache.org/contribute/release-guide/#vote-on-the-release-candidate.

> Verify default container image names in Python SDK
> --
>
> Key: BEAM-3392
> URL: https://issues.apache.org/jira/browse/BEAM-3392
> Project: Beam
>  Issue Type: Test
>  Components: runner-dataflow
>Affects Versions: 2.3.0
>Reporter: Valentyn Tymofieiev
>Assignee: Valentyn Tymofieiev
>Priority: Blocker
> Fix For: 2.3.0
>
>
> Verify naming pattern for python containers in 2.3.0 release:
> - default legacy worker harness container image: 
> dataflow.gcr.io/v1beta3/python:2.3.0
> - pinned runner harness container image: dataflow.gcr.io/v1beta3/harness:2.3.0
> - default SDK harness container name (portable pipelines): name may change, 
> exact pattern TBD.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (BEAM-3392) Verify default container image names in Python SDK

[Valentyn Tymofieiev (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-3392?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16341543#comment-16341543
 ] 

Valentyn Tymofieiev commented on BEAM-3392:
---

Yes, it's an action to perform on a branch. The intention for the Jira was to 
do a one-time check for 2.3.0, and setting fix version to 2.3.0 meant that 
someone will look into this.

I agree that a release verification spreadsheet is a cleaner way to track this. 
Feel free to add an item to the spreadsheet and remove it from 2.3.0 blockers. 
Also, we could add information about  the spreadsheet to our release 
documentation 
https://beam.apache.org/contribute/release-guide/#vote-on-the-release-candidate.

> Verify default container image names in Python SDK
> --
>
> Key: BEAM-3392
> URL: https://issues.apache.org/jira/browse/BEAM-3392
> Project: Beam
>  Issue Type: Test
>  Components: runner-dataflow
>Affects Versions: 2.3.0
>Reporter: Valentyn Tymofieiev
>Assignee: Valentyn Tymofieiev
>Priority: Blocker
> Fix For: 2.3.0
>
>
> Verify naming pattern for python containers in 2.3.0 release:
> - default legacy worker harness container image: 
> dataflow.gcr.io/v1beta3/python:2.3.0
> - pinned runner harness container image: dataflow.gcr.io/v1beta3/harness:2.3.0
> - default SDK harness container name (portable pipelines): name may change, 
> exact pattern TBD.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (BEAM-3535) Docs on Default Coder in Python are wrong

[Luke Cwik (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-3535?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16341542#comment-16341542
 ] 

Luke Cwik commented on BEAM-3535:
-

[~maximilianroos], the docs are in [https://github.com/apache/beam-site/,] do 
you want to make a PR?

> Docs on Default Coder in Python are wrong
> -
>
> Key: BEAM-3535
> URL: https://issues.apache.org/jira/browse/BEAM-3535
> Project: Beam
>  Issue Type: Bug
>  Components: website
>Reporter: Maximilian Roos
>Assignee: Reuven Lax
>Priority: Minor
>
> These are wrong, from 
> [https://beam.apache.org/documentation/programming-guide/#default-coders-and-the-coderregistry]
>  
> |str|BytesCoder|
> |bytes|StrUtf8Coder|
>  
> Here's the relevant code: 
> [https://github.com/apache/beam/blob/master/sdks/python/apache_beam/coders/typecoders.py#L89]
>  
> The docs don't seem to be in the repo? Otherwise I'd put a PR in
>  
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (BEAM-3535) Docs on Default Coder in Python are wrong

[Luke Cwik (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3535?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Luke Cwik reassigned BEAM-3535:
---

Assignee: (was: Reuven Lax)

> Docs on Default Coder in Python are wrong
> -
>
> Key: BEAM-3535
> URL: https://issues.apache.org/jira/browse/BEAM-3535
> Project: Beam
>  Issue Type: Bug
>  Components: website
>Reporter: Maximilian Roos
>Priority: Minor
>
> These are wrong, from 
> [https://beam.apache.org/documentation/programming-guide/#default-coders-and-the-coderregistry]
>  
> |str|BytesCoder|
> |bytes|StrUtf8Coder|
>  
> Here's the relevant code: 
> [https://github.com/apache/beam/blob/master/sdks/python/apache_beam/coders/typecoders.py#L89]
>  
> The docs don't seem to be in the repo? Otherwise I'd put a PR in
>  
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (BEAM-3392) Verify default container image names in Python SDK

[Kenneth Knowles (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-3392?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16341501#comment-16341501
 ] 

Kenneth Knowles commented on BEAM-3392:
---

Do you mean that this is an action that we need to perform on the branch 
itself? I don't think validation steps belong as JIRAs, but can be put into the 
release verification spreadsheet. I think it is fine for runners to have their 
own specific needs that are part of that.

> Verify default container image names in Python SDK
> --
>
> Key: BEAM-3392
> URL: https://issues.apache.org/jira/browse/BEAM-3392
> Project: Beam
>  Issue Type: Test
>  Components: runner-dataflow
>Affects Versions: 2.3.0
>Reporter: Valentyn Tymofieiev
>Assignee: Thomas Groh
>Priority: Blocker
> Fix For: 2.3.0
>
>
> Verify naming pattern for python containers in 2.3.0 release:
> - default legacy worker harness container image: 
> dataflow.gcr.io/v1beta3/python:2.3.0
> - pinned runner harness container image: dataflow.gcr.io/v1beta3/harness:2.3.0
> - default SDK harness container name (portable pipelines): name may change, 
> exact pattern TBD.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (BEAM-3392) Verify default container image names in Python SDK

[Kenneth Knowles (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3392?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kenneth Knowles reassigned BEAM-3392:
-

Assignee: Valentyn Tymofieiev  (was: Thomas Groh)

> Verify default container image names in Python SDK
> --
>
> Key: BEAM-3392
> URL: https://issues.apache.org/jira/browse/BEAM-3392
> Project: Beam
>  Issue Type: Test
>  Components: runner-dataflow
>Affects Versions: 2.3.0
>Reporter: Valentyn Tymofieiev
>Assignee: Valentyn Tymofieiev
>Priority: Blocker
> Fix For: 2.3.0
>
>
> Verify naming pattern for python containers in 2.3.0 release:
> - default legacy worker harness container image: 
> dataflow.gcr.io/v1beta3/python:2.3.0
> - pinned runner harness container image: dataflow.gcr.io/v1beta3/harness:2.3.0
> - default SDK harness container name (portable pipelines): name may change, 
> exact pattern TBD.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: beam_PostCommit_Java_MavenInstall #5769

[Apache Jenkins Server]

See 


Changes:

[zoy] Increasing BatchElements's max_batch_size to 10K

[XuMingmin] [BEAM-3526] KakfaIO support for finalizeCheckpoint() (#4481)

--
[...truncated 2.14 MB...]
2018-01-26T19:44:48.632 [INFO] Excluding commons-lang:commons-lang:jar:2.6 from 
the shaded jar.
2018-01-26T19:44:48.632 [INFO] Excluding commons-cli:commons-cli:jar:1.2 from 
the shaded jar.
2018-01-26T19:44:48.632 [INFO] Excluding log4j:log4j:jar:1.2.17 from the shaded 
jar.
2018-01-26T19:44:48.632 [INFO] Excluding 
org.apache.hadoop:hadoop-annotations:jar:2.6.0 from the shaded jar.
2018-01-26T19:44:48.632 [INFO] Excluding 
org.apache.hadoop:hadoop-yarn-api:jar:2.6.0 from the shaded jar.
2018-01-26T19:44:48.632 [INFO] Excluding 
org.apache.hadoop:hadoop-yarn-common:jar:2.6.0 from the shaded jar.
2018-01-26T19:44:48.632 [INFO] Excluding javax.xml.bind:jaxb-api:jar:2.2.2 from 
the shaded jar.
2018-01-26T19:44:48.632 [INFO] Excluding javax.xml.stream:stax-api:jar:1.0-2 
from the shaded jar.
2018-01-26T19:44:48.632 [INFO] Excluding 
com.google.inject.extensions:guice-servlet:jar:3.0 from the shaded jar.
2018-01-26T19:44:48.632 [INFO] Excluding com.google.inject:guice:jar:3.0 from 
the shaded jar.
2018-01-26T19:44:48.632 [INFO] Excluding javax.inject:javax.inject:jar:1 from 
the shaded jar.
2018-01-26T19:44:48.632 [INFO] Excluding aopalliance:aopalliance:jar:1.0 from 
the shaded jar.
2018-01-26T19:44:48.632 [INFO] Excluding com.sun.jersey:jersey-server:jar:1.9 
from the shaded jar.
2018-01-26T19:44:48.632 [INFO] Excluding asm:asm:jar:3.1 from the shaded jar.
2018-01-26T19:44:48.632 [INFO] Excluding 
com.sun.jersey.contribs:jersey-guice:jar:1.9 from the shaded jar.
2018-01-26T19:44:48.632 [INFO] Excluding jline:jline:jar:2.11 from the shaded 
jar.
2018-01-26T19:44:48.632 [INFO] Excluding org.apache.ant:ant:jar:1.9.2 from the 
shaded jar.
2018-01-26T19:44:48.632 [INFO] Excluding org.apache.ant:ant-launcher:jar:1.9.2 
from the shaded jar.
2018-01-26T19:44:48.632 [INFO] Excluding net.engio:mbassador:jar:1.1.9 from the 
shaded jar.
2018-01-26T19:44:48.632 [INFO] Excluding net.lingala.zip4j:zip4j:jar:1.3.2 from 
the shaded jar.
2018-01-26T19:44:48.632 [INFO] Excluding commons-codec:commons-codec:jar:1.10 
from the shaded jar.
2018-01-26T19:44:48.632 [INFO] Excluding 
org.apache.xbean:xbean-asm5-shaded:jar:4.3 from the shaded jar.
2018-01-26T19:44:48.632 [INFO] Excluding org.jctools:jctools-core:jar:1.1 from 
the shaded jar.
2018-01-26T19:44:48.632 [INFO] Excluding 
org.apache.beam:beam-model-pipeline:jar:2.3.0-SNAPSHOT from the shaded jar.
2018-01-26T19:44:48.632 [INFO] Excluding 
com.google.protobuf:protobuf-java:jar:3.2.0 from the shaded jar.
2018-01-26T19:44:48.632 [INFO] Excluding 
org.apache.beam:beam-sdks-java-core:jar:2.3.0-SNAPSHOT from the shaded jar.
2018-01-26T19:44:48.632 [INFO] Including com.google.guava:guava:jar:20.0 in the 
shaded jar.
2018-01-26T19:44:48.632 [INFO] Excluding 
com.github.stephenc.findbugs:findbugs-annotations:jar:1.3.9-1 from the shaded 
jar.
2018-01-26T19:44:48.632 [INFO] Excluding 
com.fasterxml.jackson.core:jackson-core:jar:2.8.9 from the shaded jar.
2018-01-26T19:44:48.632 [INFO] Excluding 
com.fasterxml.jackson.core:jackson-annotations:jar:2.8.9 from the shaded jar.
2018-01-26T19:44:48.632 [INFO] Excluding 
com.fasterxml.jackson.core:jackson-databind:jar:2.8.9 from the shaded jar.
2018-01-26T19:44:48.632 [INFO] Excluding net.bytebuddy:byte-buddy:jar:1.6.8 
from the shaded jar.
2018-01-26T19:44:48.632 [INFO] Excluding org.apache.avro:avro:jar:1.8.2 from 
the shaded jar.
2018-01-26T19:44:48.632 [INFO] Excluding 
com.thoughtworks.paranamer:paranamer:jar:2.7 from the shaded jar.
2018-01-26T19:44:48.632 [INFO] Excluding org.tukaani:xz:jar:1.5 from the shaded 
jar.
2018-01-26T19:44:48.632 [INFO] Excluding 
org.xerial.snappy:snappy-java:jar:1.1.4 from the shaded jar.
2018-01-26T19:44:48.632 [INFO] Excluding 
org.apache.commons:commons-compress:jar:1.14 from the shaded jar.
2018-01-26T19:44:48.632 [INFO] Excluding 
org.apache.beam:beam-runners-core-construction-java:jar:2.3.0-SNAPSHOT from the 
shaded jar.
2018-01-26T19:44:48.632 [INFO] Excluding 
org.apache.beam:beam-model-job-management:jar:2.3.0-SNAPSHOT from the shaded 
jar.
2018-01-26T19:44:48.632 [INFO] Excluding 
com.google.protobuf:protobuf-java-util:jar:3.2.0 from the shaded jar.
2018-01-26T19:44:48.632 [INFO] Excluding com.google.code.gson:gson:jar:2.7 from 
the shaded jar.
2018-01-26T19:44:48.632 [INFO] Excluding io.grpc:grpc-core:jar:1.2.0 from the 
shaded jar.
2018-01-26T19:44:48.632 [INFO] Excluding 
com.google.errorprone:error_prone_annotations:jar:2.0.15 from the shaded jar.
2018-01-26T19:44:48.632 [INFO] Excluding io.grpc:grpc-context:jar:1.2.0 from 
the shaded jar.
2018-01-26T19:44:48.632 [INFO] Excluding 
com.google.instrumentation:instrumentation-api:jar:0.3.0 from the shaded jar.
2018-01-26T19:44:48.632 

Build failed in Jenkins: beam_PostCommit_Java_ValidatesRunner_Spark #4002

[Apache Jenkins Server]

See 


Changes:

[zoy] Increasing BatchElements's max_batch_size to 10K

[lcwik] [BEAM-2273] Cleanup examples Maven Archetype to copy in a clean state

[XuMingmin] [BEAM-3526] KakfaIO support for finalizeCheckpoint() (#4481)

--
[...truncated 569.47 KB...]
2018-01-26T19:09:32.331 [INFO] Downloading from central: 
https://repo.maven.apache.org/maven2/org/lz4/lz4-java/1.4/lz4-java-1.4.jar
2018-01-26T19:09:32.331 [INFO] Downloading from central: 
https://repo.maven.apache.org/maven2/org/springframework/spring-core/4.3.5.RELEASE/spring-core-4.3.5.RELEASE.jar
2018-01-26T19:09:32.331 [INFO] Downloading from central: 
https://repo.maven.apache.org/maven2/org/springframework/spring-expression/4.3.5.RELEASE/spring-expression-4.3.5.RELEASE.jar
2018-01-26T19:09:32.331 [INFO] Downloading from central: 
https://repo.maven.apache.org/maven2/commons-logging/commons-logging/1.2/commons-logging-1.2.jar
2018-01-26T19:09:32.368 [INFO] Downloaded from central: 
https://repo.maven.apache.org/maven2/commons-logging/commons-logging/1.2/commons-logging-1.2.jar
 (62 kB at 1.7 MB/s)
2018-01-26T19:09:32.384 [INFO] Downloaded from central: 
https://repo.maven.apache.org/maven2/org/springframework/spring-expression/4.3.5.RELEASE/spring-expression-4.3.5.RELEASE.jar
 (263 kB at 5.0 MB/s)
2018-01-26T19:09:32.391 [INFO] Downloaded from central: 
https://repo.maven.apache.org/maven2/org/lz4/lz4-java/1.4/lz4-java-1.4.jar (370 
kB at 6.1 MB/s)
2018-01-26T19:09:32.436 [INFO] Downloaded from central: 
https://repo.maven.apache.org/maven2/org/springframework/spring-core/4.3.5.RELEASE/spring-core-4.3.5.RELEASE.jar
 (1.1 MB at 11 MB/s)
2018-01-26T19:09:32.467 [INFO] Downloaded from central: 
https://repo.maven.apache.org/maven2/org/apache/kafka/kafka-clients/1.0.0/kafka-clients-1.0.0.jar
 (1.6 MB at 12 MB/s)
2018-01-26T19:09:32.470 [INFO] 
2018-01-26T19:09:32.470 [INFO] --- maven-clean-plugin:3.0.0:clean 
(default-clean) @ beam-sdks-java-io-kafka ---
2018-01-26T19:09:32.474 [INFO] Deleting 

 (includes = [**/*.pyc, **/*.egg-info/, **/sdks/python/LICENSE, 
**/sdks/python/NOTICE, **/sdks/python/README.md, 
**/sdks/python/apache_beam/portability/api/*pb2*.*, **/sdks/python/**/*.c, 
**/sdks/python/**/*.so, **/sdks/python/**/*.egg], excludes = [])
2018-01-26T19:09:32.625 [INFO] 
2018-01-26T19:09:32.625 [INFO] --- maven-enforcer-plugin:3.0.0-M1:enforce 
(enforce) @ beam-sdks-java-io-kafka ---
2018-01-26T19:09:33.372 [INFO] 
2018-01-26T19:09:33.372 [INFO] --- maven-enforcer-plugin:3.0.0-M1:enforce 
(enforce-banned-dependencies) @ beam-sdks-java-io-kafka ---
2018-01-26T19:09:33.488 [INFO] 
2018-01-26T19:09:33.488 [INFO] --- maven-remote-resources-plugin:1.5:process 
(process-resource-bundles) @ beam-sdks-java-io-kafka ---
2018-01-26T19:09:34.034 [INFO] 
2018-01-26T19:09:34.034 [INFO] --- maven-resources-plugin:3.0.2:resources 
(default-resources) @ beam-sdks-java-io-kafka ---
2018-01-26T19:09:34.035 [INFO] Using 'UTF-8' encoding to copy filtered 
resources.
2018-01-26T19:09:34.035 [INFO] skip non existing resourceDirectory 

2018-01-26T19:09:34.035 [INFO] Copying 3 resources
2018-01-26T19:09:34.181 [INFO] 
2018-01-26T19:09:34.182 [INFO] --- maven-compiler-plugin:3.6.2:compile 
(default-compile) @ beam-sdks-java-io-kafka ---
2018-01-26T19:09:34.186 [INFO] Changes detected - recompiling the module!
2018-01-26T19:09:34.187 [INFO] Compiling 10 source files to 

2018-01-26T19:09:34.869 [INFO] 
:
 

 uses or overrides a deprecated API.
2018-01-26T19:09:34.869 [INFO] 
:
 Recompile with -Xlint:deprecation for details.
2018-01-26T19:09:34.869 [INFO] 
:
 Some input files use unchecked or unsafe operations.
2018-01-26T19:09:34.869 [INFO] 
:
 Recompile with -Xlint:unchecked for details.

Jenkins build is still unstable: beam_PostCommit_Java_ValidatesRunner_Spark #4001

[Apache Jenkins Server]

See 

[beam] branch master updated: [BEAM-3526] KakfaIO support for finalizeCheckpoint() (#4481)

[mingmxu]

This is an automated email from the ASF dual-hosted git repository.

mingmxu pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git


The following commit(s) were added to refs/heads/master by this push:
 new 1274938  [BEAM-3526] KakfaIO support for finalizeCheckpoint() (#4481)
1274938 is described below

commit 1274938bcfd1c2233f27b9b6c5fa5d10370e5ade
Author: Raghu Angadi 
AuthorDate: Fri Jan 26 10:41:05 2018 -0800

[BEAM-3526] KakfaIO support for finalizeCheckpoint() (#4481)

* KakfaIO : Add support for finalizeCheckpoint()

Adds an option to KafkaIO source to commit offsets to Kafka inside
`CheckpointMark#finalizecheckPoint()`. Kafka consumer is single
threaded and as a result actuall commit to Kafka might take upto
1 second (KAFKA_POLL_TIMEOUT). This is not necessarily bad since
it could avoid excessive commits when multiple checkpoint marks
are finalized at once by the runner.

* @AvroIgnore reader field in checkpoint mark.

* reset records to emty.

* minor

* Set enq timeout to 100 millis

* check return of offer(); remove metric check in unit test.

* Warn if auto commit is enabled.
---
 .../beam/sdk/io/kafka/KafkaCheckpointMark.java |  20 +++-
 .../java/org/apache/beam/sdk/io/kafka/KafkaIO.java | 118 ++---
 .../org/apache/beam/sdk/io/kafka/KafkaIOTest.java  |   9 +-
 3 files changed, 126 insertions(+), 21 deletions(-)

diff --git 
a/sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaCheckpointMark.java
 
b/sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaCheckpointMark.java
index 64dbad4..0856c7c 100644
--- 
a/sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaCheckpointMark.java
+++ 
b/sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaCheckpointMark.java
@@ -21,6 +21,8 @@ import com.google.common.base.Joiner;
 import java.io.IOException;
 import java.io.Serializable;
 import java.util.List;
+
+import org.apache.avro.reflect.AvroIgnore;
 import org.apache.beam.sdk.coders.AvroCoder;
 import org.apache.beam.sdk.coders.DefaultCoder;
 import org.apache.beam.sdk.io.UnboundedSource;
@@ -34,10 +36,15 @@ public class KafkaCheckpointMark implements 
UnboundedSource.CheckpointMark {
 
   private List partitions;
 
+  @AvroIgnore
+  private KafkaIO.UnboundedKafkaReader reader; // Non-null when offsets 
need to be committed.
+
   private KafkaCheckpointMark() {} // for Avro
 
-  public KafkaCheckpointMark(List partitions) {
+  public KafkaCheckpointMark(List partitions,
+ KafkaIO.UnboundedKafkaReader reader) {
 this.partitions = partitions;
+this.reader = reader;
   }
 
   public List getPartitions() {
@@ -46,10 +53,13 @@ public class KafkaCheckpointMark implements 
UnboundedSource.CheckpointMark {
 
   @Override
   public void finalizeCheckpoint() throws IOException {
-/* nothing to do */
-
-// We might want to support committing offset in Kafka for better resume 
point when the job
-// is restarted (checkpoint is not available for job restarts).
+if (reader != null) {
+  // Is it ok to commit asynchronously, or should we wait till this (or 
newer) is committed?
+  // Often multiple marks would be finalized at once, since we only need 
to finalize the latest,
+  // it is better to wait a little while. Currently maximum is delay same 
as KAFKA_POLL_TIMEOUT
+  // in the reader (1 second).
+  reader.finalizeCheckpointMarkAsync(this);
+}
   }
 
   @Override
diff --git 
a/sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaIO.java 
b/sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaIO.java
index 374dbc7..aa931e6 100644
--- a/sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaIO.java
+++ b/sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaIO.java
@@ -62,6 +62,8 @@ import java.util.concurrent.ThreadLocalRandom;
 import java.util.concurrent.TimeUnit;
 import java.util.concurrent.TimeoutException;
 import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.concurrent.atomic.AtomicReference;
+import java.util.stream.Collectors;
 import javax.annotation.Nullable;
 import org.apache.beam.sdk.annotations.Experimental;
 import org.apache.beam.sdk.coders.AtomicCoder;
@@ -302,6 +304,7 @@ public class KafkaIO {
 .setConsumerFactoryFn(Read.KAFKA_CONSUMER_FACTORY_FN)
 .setConsumerConfig(Read.DEFAULT_CONSUMER_PROPERTIES)
 .setMaxNumRecords(Long.MAX_VALUE)
+.setCommitOffsetsInFinalizeEnabled(false)
 .build();
   }
 
@@ -344,6 +347,7 @@ public class KafkaIO {
 @Nullable abstract Duration getMaxReadTime();
 
 @Nullable abstract Instant getStartReadTime();
+abstract boolean isCommitOffsetsInFinalizeEnabled();
 
 abstract Builder toBuilder();
 
@@ -364,6 +368,7 @@ public class KafkaIO {
 

[jira] [Commented] (BEAM-3392) Verify default container image names in Python SDK

[Valentyn Tymofieiev (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-3392?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16341412#comment-16341412
 ] 

Valentyn Tymofieiev commented on BEAM-3392:
---

Yes, it's a step a Dataflow maintainer should perform when we make a 2.3.0 
release branch. It's not a blocker for cutting a branch.

> Verify default container image names in Python SDK
> --
>
> Key: BEAM-3392
> URL: https://issues.apache.org/jira/browse/BEAM-3392
> Project: Beam
>  Issue Type: Test
>  Components: runner-dataflow
>Affects Versions: 2.3.0
>Reporter: Valentyn Tymofieiev
>Assignee: Thomas Groh
>Priority: Blocker
> Fix For: 2.3.0
>
>
> Verify naming pattern for python containers in 2.3.0 release:
> - default legacy worker harness container image: 
> dataflow.gcr.io/v1beta3/python:2.3.0
> - pinned runner harness container image: dataflow.gcr.io/v1beta3/harness:2.3.0
> - default SDK harness container name (portable pipelines): name may change, 
> exact pattern TBD.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[beam] 01/01: Merge pull request #4430 from zoyahav/master

[altay]

This is an automated email from the ASF dual-hosted git repository.

altay pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git

commit a8239dc70d55e98b7586fa5525564c72a5623e00
Merge: a16b7a9 64b7d99
Author: Ahmet Altay 
AuthorDate: Fri Jan 26 10:30:43 2018 -0800

Merge pull request #4430 from zoyahav/master

Increasing BatchElements's max_batch_size to 10K

 sdks/python/apache_beam/transforms/util.py | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)


-- 
To stop receiving notification emails like this one, please contact
al...@apache.org.

[beam] branch master updated (a16b7a9 -> a8239dc)

[altay]

This is an automated email from the ASF dual-hosted git repository.

altay pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from a16b7a9  [BEAM-2273] Cleanup examples Maven Archetype to copy in a 
clean state
 add 64b7d99  Increasing BatchElements's max_batch_size to 10K
 new a8239dc  Merge pull request #4430 from zoyahav/master

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 sdks/python/apache_beam/transforms/util.py | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

-- 
To stop receiving notification emails like this one, please contact
al...@apache.org.

[jira] [Comment Edited] (BEAM-2273) mvn clean doesn't fully clean up archetypes.

[Luke Cwik (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2273?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16341380#comment-16341380
 ] 

Luke Cwik edited comment on BEAM-2273 at 1/26/18 6:17 PM:
--

All that is left is to also make sure that *sdks/java/maven-archetypes/starter* 
also is cleaned up.


was (Author: lcwik):
All that is left is to also make sure that `sdks/java/maven-archetypes/starter` 
also is cleaned up.

> mvn clean doesn't fully clean up archetypes.
> 
>
> Key: BEAM-2273
> URL: https://issues.apache.org/jira/browse/BEAM-2273
> Project: Beam
>  Issue Type: Bug
>  Components: build-system
>Reporter: Jason Kuster
>Assignee: Jean-Baptiste Onofré
>Priority: Major
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (BEAM-2273) mvn clean doesn't fully clean up archetypes.

[Luke Cwik (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2273?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16341380#comment-16341380
 ] 

Luke Cwik commented on BEAM-2273:
-

All that is left is to also make sure that `sdks/java/maven-archetypes/starter` 
also is cleaned up.

> mvn clean doesn't fully clean up archetypes.
> 
>
> Key: BEAM-2273
> URL: https://issues.apache.org/jira/browse/BEAM-2273
> Project: Beam
>  Issue Type: Bug
>  Components: build-system
>Reporter: Jason Kuster
>Assignee: Jean-Baptiste Onofré
>Priority: Major
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[beam] branch master updated: [BEAM-2273] Cleanup examples Maven Archetype to copy in a clean state

[lcwik]

This is an automated email from the ASF dual-hosted git repository.

lcwik pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git


The following commit(s) were added to refs/heads/master by this push:
 new a16b7a9  [BEAM-2273] Cleanup examples Maven Archetype to copy in a 
clean state
a16b7a9 is described below

commit a16b7a951c242580cd666728664953f4b2b3e204
Author: Jean-Baptiste Onofré 
AuthorDate: Thu Jan 25 18:13:18 2018 +0100

[BEAM-2273] Cleanup examples Maven Archetype to copy in a clean state
---
 sdks/java/maven-archetypes/examples/pom.xml | 16 
 1 file changed, 16 insertions(+)

diff --git a/sdks/java/maven-archetypes/examples/pom.xml 
b/sdks/java/maven-archetypes/examples/pom.xml
index e658c3b..fab9f73 100644
--- a/sdks/java/maven-archetypes/examples/pom.xml
+++ b/sdks/java/maven-archetypes/examples/pom.xml
@@ -91,6 +91,22 @@
   
 
   
+  
+org.apache.maven.plugins
+maven-clean-plugin
+
+  
+
+  src/main/resources/archetype-resources
+  
+src/**/*
+src
+  
+  false
+
+  
+
+  
 
   
 

-- 
To stop receiving notification emails like this one, please contact
lc...@apache.org.

Jenkins build is still unstable: beam_PostCommit_Java_ValidatesRunner_Spark #4000

[Apache Jenkins Server]

See 

Build failed in Jenkins: beam_PerformanceTests_TFRecordIOIT #62

[Apache Jenkins Server]

See 


Changes:

[tgroh] Retrieve Environments from PTransforms

[iemejia] Remove unneeded profile for javadoc on Java 8

[iemejia] Remove unneeded explicit Java 8 references on maven-compiler-plugin

[iemejia] Fix doc error on hadoop-input-format ITs after move to Java 8 only 
tests

[iemejia] Remove references to non-existent examples:java8 module in gradle

[iemejia] Remove references to java 7/8 only examples from the README

[iemejia] Remove some comments on Java 7/8 only stuff that don't make sense

--
Started by timer
[EnvInject] - Loading node environment variables.
Building remotely on beam6 (beam) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* 
 > +refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/*
 > git rev-parse origin/master^{commit} # timeout=10
Checking out Revision d4a55d9f485a06a9fc0933884c36d38af3c69d3d (origin/master)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f d4a55d9f485a06a9fc0933884c36d38af3c69d3d
Commit message: "Merge pull request #4469"
 > git rev-list 87670e6f525f3a9e51f6603f072410f86be48447 # timeout=10
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
[EnvInject] - Executing scripts and injecting environment variables after the 
SCM step.
[EnvInject] - Injecting as environment variables the properties content 
SPARK_LOCAL_IP=127.0.0.1

[EnvInject] - Variables injected successfully.
[beam_PerformanceTests_TFRecordIOIT] $ /bin/bash -xe 
/tmp/jenkins7819830588121155430.sh
+ rm -rf PerfKitBenchmarker
[beam_PerformanceTests_TFRecordIOIT] $ /bin/bash -xe 
/tmp/jenkins64803583749974492.sh
+ rm -rf .env
[beam_PerformanceTests_TFRecordIOIT] $ /bin/bash -xe 
/tmp/jenkins3135516927771430401.sh
+ virtualenv .env --system-site-packages
New python executable in .env/bin/python
Installing setuptools, pip...done.
[beam_PerformanceTests_TFRecordIOIT] $ /bin/bash -xe 
/tmp/jenkins6499499629155285698.sh
+ git clone https://github.com/GoogleCloudPlatform/PerfKitBenchmarker.git
Cloning into 'PerfKitBenchmarker'...
[beam_PerformanceTests_TFRecordIOIT] $ /bin/bash -xe 
/tmp/jenkins2223274130981627251.sh
+ .env/bin/pip install -r PerfKitBenchmarker/requirements.txt
Requirement already satisfied (use --upgrade to upgrade): absl-py in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 14))
Requirement already satisfied (use --upgrade to upgrade): jinja2>=2.7 in 
/usr/local/lib/python2.7/dist-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 15))
Requirement already satisfied (use --upgrade to upgrade): setuptools in 
./.env/lib/python2.7/site-packages (from -r PerfKitBenchmarker/requirements.txt 
(line 16))
Requirement already satisfied (use --upgrade to upgrade): 
colorlog[windows]==2.6.0 in /home/jenkins/.local/lib/python2.7/site-packages 
(from -r PerfKitBenchmarker/requirements.txt (line 17))
  Installing extra requirements: 'windows'
Requirement already satisfied (use --upgrade to upgrade): blinker>=1.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 18))
Requirement already satisfied (use --upgrade to upgrade): futures>=3.0.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 19))
Requirement already satisfied (use --upgrade to upgrade): PyYAML==3.12 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 20))
Requirement already satisfied (use --upgrade to upgrade): pint>=0.7 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 21))
Requirement already satisfied (use --upgrade to upgrade): numpy in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 22))
Requirement already satisfied (use --upgrade to upgrade): functools32 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 23))
Requirement already satisfied (use --upgrade to upgrade): contextlib2>=0.5.1 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 24))
Requirement already satisfied (use --upgrade to upgrade): pywinrm in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 25))

Build failed in Jenkins: beam_PerformanceTests_TextIOIT #70

[Apache Jenkins Server]

See 


Changes:

[tgroh] Retrieve Environments from PTransforms

[iemejia] Remove unneeded profile for javadoc on Java 8

[iemejia] Remove unneeded explicit Java 8 references on maven-compiler-plugin

[iemejia] Fix doc error on hadoop-input-format ITs after move to Java 8 only 
tests

[iemejia] Remove references to non-existent examples:java8 module in gradle

[iemejia] Remove references to java 7/8 only examples from the README

[iemejia] Remove some comments on Java 7/8 only stuff that don't make sense

--
Started by timer
[EnvInject] - Loading node environment variables.
Building remotely on beam4 (beam) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* 
 > +refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/*
 > git rev-parse origin/master^{commit} # timeout=10
Checking out Revision d4a55d9f485a06a9fc0933884c36d38af3c69d3d (origin/master)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f d4a55d9f485a06a9fc0933884c36d38af3c69d3d
Commit message: "Merge pull request #4469"
 > git rev-list 87670e6f525f3a9e51f6603f072410f86be48447 # timeout=10
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
[EnvInject] - Executing scripts and injecting environment variables after the 
SCM step.
[EnvInject] - Injecting as environment variables the properties content 
SPARK_LOCAL_IP=127.0.0.1

[EnvInject] - Variables injected successfully.
[beam_PerformanceTests_TextIOIT] $ /bin/bash -xe 
/tmp/jenkins5197036074934553989.sh
+ rm -rf PerfKitBenchmarker
[beam_PerformanceTests_TextIOIT] $ /bin/bash -xe 
/tmp/jenkins1771939912572952884.sh
+ rm -rf .env
[beam_PerformanceTests_TextIOIT] $ /bin/bash -xe 
/tmp/jenkins5222068190458999431.sh
+ virtualenv .env --system-site-packages
New python executable in .env/bin/python
Installing setuptools, pip...done.
[beam_PerformanceTests_TextIOIT] $ /bin/bash -xe 
/tmp/jenkins7968763689483846002.sh
+ git clone https://github.com/GoogleCloudPlatform/PerfKitBenchmarker.git
Cloning into 'PerfKitBenchmarker'...
[beam_PerformanceTests_TextIOIT] $ /bin/bash -xe 
/tmp/jenkins7289848688440660400.sh
+ .env/bin/pip install -r PerfKitBenchmarker/requirements.txt
Requirement already satisfied (use --upgrade to upgrade): absl-py in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 14))
Requirement already satisfied (use --upgrade to upgrade): jinja2>=2.7 in 
/usr/local/lib/python2.7/dist-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 15))
Requirement already satisfied (use --upgrade to upgrade): setuptools in 
./.env/lib/python2.7/site-packages (from -r PerfKitBenchmarker/requirements.txt 
(line 16))
Requirement already satisfied (use --upgrade to upgrade): 
colorlog[windows]==2.6.0 in /home/jenkins/.local/lib/python2.7/site-packages 
(from -r PerfKitBenchmarker/requirements.txt (line 17))
  Installing extra requirements: 'windows'
Requirement already satisfied (use --upgrade to upgrade): blinker>=1.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 18))
Requirement already satisfied (use --upgrade to upgrade): futures>=3.0.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 19))
Requirement already satisfied (use --upgrade to upgrade): PyYAML==3.12 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 20))
Requirement already satisfied (use --upgrade to upgrade): pint>=0.7 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 21))
Requirement already satisfied (use --upgrade to upgrade): numpy in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 22))
Requirement already satisfied (use --upgrade to upgrade): functools32 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 23))
Requirement already satisfied (use --upgrade to upgrade): contextlib2>=0.5.1 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 24))
Requirement already satisfied (use --upgrade to upgrade): pywinrm in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 25))
Cleaning up...

Build failed in Jenkins: beam_PerformanceTests_Spark #1282

[Apache Jenkins Server]

See 


Changes:

[tgroh] Retrieve Environments from PTransforms

[iemejia] Remove unneeded profile for javadoc on Java 8

[iemejia] Remove unneeded explicit Java 8 references on maven-compiler-plugin

[iemejia] Fix doc error on hadoop-input-format ITs after move to Java 8 only 
tests

[iemejia] Remove references to non-existent examples:java8 module in gradle

[iemejia] Remove references to java 7/8 only examples from the README

[iemejia] Remove some comments on Java 7/8 only stuff that don't make sense

--
Started by timer
[EnvInject] - Loading node environment variables.
Building remotely on beam6 (beam) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* 
 > +refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/*
 > git rev-parse origin/master^{commit} # timeout=10
Checking out Revision d4a55d9f485a06a9fc0933884c36d38af3c69d3d (origin/master)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f d4a55d9f485a06a9fc0933884c36d38af3c69d3d
Commit message: "Merge pull request #4469"
 > git rev-list 87670e6f525f3a9e51f6603f072410f86be48447 # timeout=10
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
[EnvInject] - Executing scripts and injecting environment variables after the 
SCM step.
[EnvInject] - Injecting as environment variables the properties content 
SPARK_LOCAL_IP=127.0.0.1

[EnvInject] - Variables injected successfully.
[beam_PerformanceTests_Spark] $ /bin/bash -xe /tmp/jenkins2122607179555377579.sh
+ rm -rf PerfKitBenchmarker
[beam_PerformanceTests_Spark] $ /bin/bash -xe /tmp/jenkins2795165434589182382.sh
+ rm -rf .env
[beam_PerformanceTests_Spark] $ /bin/bash -xe /tmp/jenkins2320738825781312042.sh
+ virtualenv .env --system-site-packages
New python executable in .env/bin/python
Installing setuptools, pip...done.
[beam_PerformanceTests_Spark] $ /bin/bash -xe /tmp/jenkins7600526568084628795.sh
+ git clone https://github.com/GoogleCloudPlatform/PerfKitBenchmarker.git
Cloning into 'PerfKitBenchmarker'...
[beam_PerformanceTests_Spark] $ /bin/bash -xe /tmp/jenkins1005220369100925484.sh
+ .env/bin/pip install -r PerfKitBenchmarker/requirements.txt
Requirement already satisfied (use --upgrade to upgrade): absl-py in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 14))
Requirement already satisfied (use --upgrade to upgrade): jinja2>=2.7 in 
/usr/local/lib/python2.7/dist-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 15))
Requirement already satisfied (use --upgrade to upgrade): setuptools in 
./.env/lib/python2.7/site-packages (from -r PerfKitBenchmarker/requirements.txt 
(line 16))
Requirement already satisfied (use --upgrade to upgrade): 
colorlog[windows]==2.6.0 in /home/jenkins/.local/lib/python2.7/site-packages 
(from -r PerfKitBenchmarker/requirements.txt (line 17))
  Installing extra requirements: 'windows'
Requirement already satisfied (use --upgrade to upgrade): blinker>=1.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 18))
Requirement already satisfied (use --upgrade to upgrade): futures>=3.0.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 19))
Requirement already satisfied (use --upgrade to upgrade): PyYAML==3.12 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 20))
Requirement already satisfied (use --upgrade to upgrade): pint>=0.7 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 21))
Requirement already satisfied (use --upgrade to upgrade): numpy in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 22))
Requirement already satisfied (use --upgrade to upgrade): functools32 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 23))
Requirement already satisfied (use --upgrade to upgrade): contextlib2>=0.5.1 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 24))
Requirement already satisfied (use --upgrade to upgrade): pywinrm in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 25))
Cleaning up...
[beam_PerformanceTests_Spark] $ /bin/bash 

Build failed in Jenkins: beam_PerformanceTests_Python #838

[Apache Jenkins Server]

See 


Changes:

[tgroh] Retrieve Environments from PTransforms

[iemejia] Remove unneeded profile for javadoc on Java 8

[iemejia] Remove unneeded explicit Java 8 references on maven-compiler-plugin

[iemejia] Fix doc error on hadoop-input-format ITs after move to Java 8 only 
tests

[iemejia] Remove references to non-existent examples:java8 module in gradle

[iemejia] Remove references to java 7/8 only examples from the README

[iemejia] Remove some comments on Java 7/8 only stuff that don't make sense

--
Started by timer
[EnvInject] - Loading node environment variables.
Building remotely on beam6 (beam) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* 
 > +refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/*
 > git rev-parse origin/master^{commit} # timeout=10
Checking out Revision d4a55d9f485a06a9fc0933884c36d38af3c69d3d (origin/master)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f d4a55d9f485a06a9fc0933884c36d38af3c69d3d
Commit message: "Merge pull request #4469"
 > git rev-list 87670e6f525f3a9e51f6603f072410f86be48447 # timeout=10
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
[EnvInject] - Executing scripts and injecting environment variables after the 
SCM step.
[EnvInject] - Injecting as environment variables the properties content 
SPARK_LOCAL_IP=127.0.0.1

[EnvInject] - Variables injected successfully.
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins9122977290688440520.sh
+ rm -rf PerfKitBenchmarker
[beam_PerformanceTests_Python] $ /bin/bash -xe /tmp/jenkins38554171917008379.sh
+ rm -rf .env
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins7999034665445874876.sh
+ virtualenv .env --system-site-packages
New python executable in .env/bin/python
Installing setuptools, pip...done.
[beam_PerformanceTests_Python] $ /bin/bash -xe /tmp/jenkins287907518671620106.sh
+ git clone https://github.com/GoogleCloudPlatform/PerfKitBenchmarker.git
Cloning into 'PerfKitBenchmarker'...
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins9121776821158601154.sh
+ .env/bin/pip install -r PerfKitBenchmarker/requirements.txt
Requirement already satisfied (use --upgrade to upgrade): absl-py in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 14))
Requirement already satisfied (use --upgrade to upgrade): jinja2>=2.7 in 
/usr/local/lib/python2.7/dist-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 15))
Requirement already satisfied (use --upgrade to upgrade): setuptools in 
./.env/lib/python2.7/site-packages (from -r PerfKitBenchmarker/requirements.txt 
(line 16))
Requirement already satisfied (use --upgrade to upgrade): 
colorlog[windows]==2.6.0 in /home/jenkins/.local/lib/python2.7/site-packages 
(from -r PerfKitBenchmarker/requirements.txt (line 17))
  Installing extra requirements: 'windows'
Requirement already satisfied (use --upgrade to upgrade): blinker>=1.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 18))
Requirement already satisfied (use --upgrade to upgrade): futures>=3.0.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 19))
Requirement already satisfied (use --upgrade to upgrade): PyYAML==3.12 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 20))
Requirement already satisfied (use --upgrade to upgrade): pint>=0.7 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 21))
Requirement already satisfied (use --upgrade to upgrade): numpy in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 22))
Requirement already satisfied (use --upgrade to upgrade): functools32 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 23))
Requirement already satisfied (use --upgrade to upgrade): contextlib2>=0.5.1 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 24))
Requirement already satisfied (use --upgrade to upgrade): pywinrm in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 25))
Cleaning up...
[beam_PerformanceTests_Python] $ 

Build failed in Jenkins: beam_PerformanceTests_Compressed_TextIOIT #62

[Apache Jenkins Server]

See 


Changes:

[tgroh] Retrieve Environments from PTransforms

[iemejia] Remove unneeded profile for javadoc on Java 8

[iemejia] Remove unneeded explicit Java 8 references on maven-compiler-plugin

[iemejia] Fix doc error on hadoop-input-format ITs after move to Java 8 only 
tests

[iemejia] Remove references to non-existent examples:java8 module in gradle

[iemejia] Remove references to java 7/8 only examples from the README

[iemejia] Remove some comments on Java 7/8 only stuff that don't make sense

--
Started by timer
[EnvInject] - Loading node environment variables.
Building remotely on beam6 (beam) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* 
 > +refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/*
 > git rev-parse origin/master^{commit} # timeout=10
Checking out Revision d4a55d9f485a06a9fc0933884c36d38af3c69d3d (origin/master)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f d4a55d9f485a06a9fc0933884c36d38af3c69d3d
Commit message: "Merge pull request #4469"
 > git rev-list 87670e6f525f3a9e51f6603f072410f86be48447 # timeout=10
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
[EnvInject] - Executing scripts and injecting environment variables after the 
SCM step.
[EnvInject] - Injecting as environment variables the properties content 
SPARK_LOCAL_IP=127.0.0.1

[EnvInject] - Variables injected successfully.
[beam_PerformanceTests_Compressed_TextIOIT] $ /bin/bash -xe 
/tmp/jenkins863503945950506209.sh
+ rm -rf PerfKitBenchmarker
[beam_PerformanceTests_Compressed_TextIOIT] $ /bin/bash -xe 
/tmp/jenkins2524961945300356562.sh
+ rm -rf .env
[beam_PerformanceTests_Compressed_TextIOIT] $ /bin/bash -xe 
/tmp/jenkins1299840921073537292.sh
+ virtualenv .env --system-site-packages
New python executable in .env/bin/python
Installing setuptools, pip...done.
[beam_PerformanceTests_Compressed_TextIOIT] $ /bin/bash -xe 
/tmp/jenkins5776208691699343440.sh
+ git clone https://github.com/GoogleCloudPlatform/PerfKitBenchmarker.git
Cloning into 'PerfKitBenchmarker'...
[beam_PerformanceTests_Compressed_TextIOIT] $ /bin/bash -xe 
/tmp/jenkins6385716268513662226.sh
+ .env/bin/pip install -r PerfKitBenchmarker/requirements.txt
Requirement already satisfied (use --upgrade to upgrade): absl-py in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 14))
Requirement already satisfied (use --upgrade to upgrade): jinja2>=2.7 in 
/usr/local/lib/python2.7/dist-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 15))
Requirement already satisfied (use --upgrade to upgrade): setuptools in 
./.env/lib/python2.7/site-packages (from -r PerfKitBenchmarker/requirements.txt 
(line 16))
Requirement already satisfied (use --upgrade to upgrade): 
colorlog[windows]==2.6.0 in /home/jenkins/.local/lib/python2.7/site-packages 
(from -r PerfKitBenchmarker/requirements.txt (line 17))
  Installing extra requirements: 'windows'
Requirement already satisfied (use --upgrade to upgrade): blinker>=1.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 18))
Requirement already satisfied (use --upgrade to upgrade): futures>=3.0.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 19))
Requirement already satisfied (use --upgrade to upgrade): PyYAML==3.12 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 20))
Requirement already satisfied (use --upgrade to upgrade): pint>=0.7 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 21))
Requirement already satisfied (use --upgrade to upgrade): numpy in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 22))
Requirement already satisfied (use --upgrade to upgrade): functools32 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 23))
Requirement already satisfied (use --upgrade to upgrade): contextlib2>=0.5.1 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 24))
Requirement already satisfied (use --upgrade to upgrade): pywinrm in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 

Build failed in Jenkins: beam_PerformanceTests_AvroIOIT #63

[Apache Jenkins Server]

See 


Changes:

[tgroh] Retrieve Environments from PTransforms

[iemejia] Remove unneeded profile for javadoc on Java 8

[iemejia] Remove unneeded explicit Java 8 references on maven-compiler-plugin

[iemejia] Fix doc error on hadoop-input-format ITs after move to Java 8 only 
tests

[iemejia] Remove references to non-existent examples:java8 module in gradle

[iemejia] Remove references to java 7/8 only examples from the README

[iemejia] Remove some comments on Java 7/8 only stuff that don't make sense

--
Started by timer
[EnvInject] - Loading node environment variables.
Building remotely on beam6 (beam) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* 
 > +refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/*
 > git rev-parse origin/master^{commit} # timeout=10
Checking out Revision d4a55d9f485a06a9fc0933884c36d38af3c69d3d (origin/master)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f d4a55d9f485a06a9fc0933884c36d38af3c69d3d
Commit message: "Merge pull request #4469"
 > git rev-list 87670e6f525f3a9e51f6603f072410f86be48447 # timeout=10
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
[EnvInject] - Executing scripts and injecting environment variables after the 
SCM step.
[EnvInject] - Injecting as environment variables the properties content 
SPARK_LOCAL_IP=127.0.0.1

[EnvInject] - Variables injected successfully.
[beam_PerformanceTests_AvroIOIT] $ /bin/bash -xe 
/tmp/jenkins909833241774693659.sh
+ rm -rf PerfKitBenchmarker
[beam_PerformanceTests_AvroIOIT] $ /bin/bash -xe 
/tmp/jenkins1563270274085246773.sh
+ rm -rf .env
[beam_PerformanceTests_AvroIOIT] $ /bin/bash -xe 
/tmp/jenkins2037368446552962042.sh
+ virtualenv .env --system-site-packages
New python executable in .env/bin/python
Installing setuptools, pip...done.
[beam_PerformanceTests_AvroIOIT] $ /bin/bash -xe 
/tmp/jenkins5845803113829001157.sh
+ git clone https://github.com/GoogleCloudPlatform/PerfKitBenchmarker.git
Cloning into 'PerfKitBenchmarker'...
[beam_PerformanceTests_AvroIOIT] $ /bin/bash -xe 
/tmp/jenkins1161650726772159021.sh
+ .env/bin/pip install -r PerfKitBenchmarker/requirements.txt
Requirement already satisfied (use --upgrade to upgrade): absl-py in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 14))
Requirement already satisfied (use --upgrade to upgrade): jinja2>=2.7 in 
/usr/local/lib/python2.7/dist-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 15))
Requirement already satisfied (use --upgrade to upgrade): setuptools in 
./.env/lib/python2.7/site-packages (from -r PerfKitBenchmarker/requirements.txt 
(line 16))
Requirement already satisfied (use --upgrade to upgrade): 
colorlog[windows]==2.6.0 in /home/jenkins/.local/lib/python2.7/site-packages 
(from -r PerfKitBenchmarker/requirements.txt (line 17))
  Installing extra requirements: 'windows'
Requirement already satisfied (use --upgrade to upgrade): blinker>=1.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 18))
Requirement already satisfied (use --upgrade to upgrade): futures>=3.0.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 19))
Requirement already satisfied (use --upgrade to upgrade): PyYAML==3.12 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 20))
Requirement already satisfied (use --upgrade to upgrade): pint>=0.7 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 21))
Requirement already satisfied (use --upgrade to upgrade): numpy in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 22))
Requirement already satisfied (use --upgrade to upgrade): functools32 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 23))
Requirement already satisfied (use --upgrade to upgrade): contextlib2>=0.5.1 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 24))
Requirement already satisfied (use --upgrade to upgrade): pywinrm in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 25))
Cleaning up...

[beam] 01/01: Merge pull request #4469

[tgroh]

This is an automated email from the ASF dual-hosted git repository.

tgroh pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git

commit d4a55d9f485a06a9fc0933884c36d38af3c69d3d
Merge: ddce6b6 0ee69fe
Author: Thomas Groh 
AuthorDate: Fri Jan 26 09:35:56 2018 -0800

Merge pull request #4469

Retrieve Environments from PTransforms

 .../core/construction/CombineTranslation.java  |   3 +-
 .../runners/core/construction/Environments.java|  72 ++-
 .../core/construction/ParDoTranslation.java|  45 ++---
 .../construction/WindowingStrategyTranslation.java |  12 +-
 .../core/construction/WriteFilesTranslation.java   |   3 +-
 .../core/construction/EnvironmentsTest.java| 224 +
 .../dataflow/PrimitiveParDoSingleFactory.java  |  13 +-
 7 files changed, 333 insertions(+), 39 deletions(-)

-- 
To stop receiving notification emails like this one, please contact
tg...@apache.org.

[beam] branch master updated (ddce6b6 -> d4a55d9)

[tgroh]

This is an automated email from the ASF dual-hosted git repository.

tgroh pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from ddce6b6  Merge pull request #4501: Some missing fixes for the Java 8 
migration
 add 0ee69fe  Retrieve Environments from PTransforms
 new d4a55d9  Merge pull request #4469

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 .../core/construction/CombineTranslation.java  |   3 +-
 .../runners/core/construction/Environments.java|  72 ++-
 .../core/construction/ParDoTranslation.java|  45 ++---
 .../construction/WindowingStrategyTranslation.java |  12 +-
 .../core/construction/WriteFilesTranslation.java   |   3 +-
 .../core/construction/EnvironmentsTest.java| 224 +
 .../dataflow/PrimitiveParDoSingleFactory.java  |  13 +-
 7 files changed, 333 insertions(+), 39 deletions(-)
 create mode 100644 
runners/core-construction-java/src/test/java/org/apache/beam/runners/core/construction/EnvironmentsTest.java

-- 
To stop receiving notification emails like this one, please contact
tg...@apache.org.

Jenkins build is still unstable: beam_PostCommit_Java_ValidatesRunner_Spark #3999

[Apache Jenkins Server]

See 

[beam] branch master updated (87670e6 -> ddce6b6)

[kenn]

This is an automated email from the ASF dual-hosted git repository.

kenn pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from 87670e6  Merge pull request #4495: Use non-deprecated version of 
Futures.transform
 add ed28a6c  Remove unneeded profile for javadoc on Java 8
 add 1cbc7e7  Remove unneeded explicit Java 8 references on 
maven-compiler-plugin
 add 29f1bbf  Fix doc error on hadoop-input-format ITs after move to Java 8 
only tests
 add 77b3b81  Remove references to non-existent examples:java8 module in 
gradle
 add 72cb3ab  Remove references to java 7/8 only examples from the README
 add c716f8a  Remove some comments on Java 7/8 only stuff that don't make 
sense anymore
 new ddce6b6  Merge pull request #4501: Some missing fixes for the Java 8 
migration

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 examples/java/README.md | 10 +++---
 pom.xml | 12 +---
 runners/java-fn-execution/pom.xml   | 14 --
 runners/pom.xml |  9 -
 .../beam/sdk/transforms/reflect/DoFnSignatures.java |  6 ++
 .../org/apache/beam/sdk/transforms/CombineTest.java |  4 
 sdks/java/fn-execution/pom.xml  | 14 --
 sdks/java/harness/pom.xml   | 17 -
 .../sdk/io/hadoop/inputformat/HIFIOCassandraIT.java |  7 ---
 .../beam/sdk/io/hadoop/inputformat/HIFIOElasticIT.java  |  7 ---
 sdks/java/java8tests/pom.xml| 12 
 sdks/java/javadoc/ant.xml   |  2 +-
 sdks/java/pom.xml   | 14 --
 settings.gradle |  1 -
 14 files changed, 15 insertions(+), 114 deletions(-)

-- 
To stop receiving notification emails like this one, please contact
k...@apache.org.

[beam] 01/01: Merge pull request #4501: Some missing fixes for the Java 8 migration

[kenn]

This is an automated email from the ASF dual-hosted git repository.

kenn pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git

commit ddce6b61a98970ce95160cf525f5f1defb9e6ce5
Merge: 87670e6 c716f8a
Author: Kenn Knowles 
AuthorDate: Fri Jan 26 07:33:41 2018 -0800

Merge pull request #4501: Some missing fixes for the Java 8 migration

 examples/java/README.md | 10 +++---
 pom.xml | 12 +---
 runners/java-fn-execution/pom.xml   | 14 --
 runners/pom.xml |  9 -
 .../beam/sdk/transforms/reflect/DoFnSignatures.java |  6 ++
 .../org/apache/beam/sdk/transforms/CombineTest.java |  4 
 sdks/java/fn-execution/pom.xml  | 14 --
 sdks/java/harness/pom.xml   | 17 -
 .../sdk/io/hadoop/inputformat/HIFIOCassandraIT.java |  7 ---
 .../beam/sdk/io/hadoop/inputformat/HIFIOElasticIT.java  |  7 ---
 sdks/java/java8tests/pom.xml| 12 
 sdks/java/javadoc/ant.xml   |  2 +-
 sdks/java/pom.xml   | 14 --
 settings.gradle |  1 -
 14 files changed, 15 insertions(+), 114 deletions(-)

-- 
To stop receiving notification emails like this one, please contact
k...@apache.org.

[jira] [Commented] (BEAM-3087) Extend lock scope in Flink UnboundedSourceWrapper

[Aljoscha Krettek (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-3087?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16341059#comment-16341059
 ] 

Aljoscha Krettek commented on BEAM-3087:


There was one user that had a problem because of this but I didn't hear about 
it recently, so I think it's ok to push it back.

> Extend lock scope in Flink UnboundedSourceWrapper
> -
>
> Key: BEAM-3087
> URL: https://issues.apache.org/jira/browse/BEAM-3087
> Project: Beam
>  Issue Type: Bug
>  Components: runner-flink
>Reporter: Aljoscha Krettek
>Priority: Critical
> Fix For: 2.3.0
>
>
> In {{UnboundedSourceWrapper}} the lock scope is not big enough: we 
> synchronise in {{emitElement()}} but should instead synchronise inside the 
> reader loop in {{run()}} because the {{Source}} interface does not allow 
> concurrent calls.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Comment Edited] (BEAM-3494) Snapshot state of aggregated data of apache beam project is not maintained in flink's checkpointing

[Aljoscha Krettek (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-3494?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16341057#comment-16341057
 ] 

Aljoscha Krettek edited comment on BEAM-3494 at 1/26/18 1:32 PM:
-

How are you enabling checkpointing? Also, could you please reformat your 
posting to put the code inside code tags to make it more readable?


was (Author: aljoscha):
How are you enabling checkpointing? Also, could you please reformat your 
posting to put the code between code tags, like this
{code:java}
{code}
your code ...
{code}{code}
to make it more readable?

> Snapshot state of aggregated data of apache beam project is not maintained in 
> flink's checkpointing 
> 
>
> Key: BEAM-3494
> URL: https://issues.apache.org/jira/browse/BEAM-3494
> Project: Beam
>  Issue Type: Bug
>  Components: runner-flink
>Reporter: suganya
>Priority: Major
>
> We have a beam project which consumes events from kafka,does a groupby in a 
> time window(5 mins),after window elapses it pushes the events to downstream 
> for merge.This project is deployed using flink ,we have enabled checkpointing 
> to recover from failed state.
> (windowsize: 5mins , checkpointingInterval: 5mins,state.backend: filesystem)
> Offsets from kafka get checkpointed every 5 
> mins(checkpointingInterval).Before finishing the entire DAG(groupBy and 
> merge) , events offsets are getting checkpointed.So incase of any restart 
> from task-manager ,new task gets started from last successful checkpoint ,but 
> we could'nt able to get the aggregated snapshot data(data from groupBy task) 
> from the persisted checkpoint.
> Able to retrieve the last successful checkpointed offset from kafka ,but 
> couldnt able to get last aggregated data till checkpointing.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (BEAM-3494) Snapshot state of aggregated data of apache beam project is not maintained in flink's checkpointing

[Aljoscha Krettek (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-3494?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16341057#comment-16341057
 ] 

Aljoscha Krettek commented on BEAM-3494:


How are you enabling checkpointing? Also, could you please reformat your 
posting to put the code between code tags, like this
{code:java}
{code}
your code ...
{code}{code}
to make it more readable?

> Snapshot state of aggregated data of apache beam project is not maintained in 
> flink's checkpointing 
> 
>
> Key: BEAM-3494
> URL: https://issues.apache.org/jira/browse/BEAM-3494
> Project: Beam
>  Issue Type: Bug
>  Components: runner-flink
>Reporter: suganya
>Priority: Major
>
> We have a beam project which consumes events from kafka,does a groupby in a 
> time window(5 mins),after window elapses it pushes the events to downstream 
> for merge.This project is deployed using flink ,we have enabled checkpointing 
> to recover from failed state.
> (windowsize: 5mins , checkpointingInterval: 5mins,state.backend: filesystem)
> Offsets from kafka get checkpointed every 5 
> mins(checkpointingInterval).Before finishing the entire DAG(groupBy and 
> merge) , events offsets are getting checkpointed.So incase of any restart 
> from task-manager ,new task gets started from last successful checkpoint ,but 
> we could'nt able to get the aggregated snapshot data(data from groupBy task) 
> from the persisted checkpoint.
> Able to retrieve the last successful checkpointed offset from kafka ,but 
> couldnt able to get last aggregated data till checkpointing.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

***UNCHECKED*** [jira] [Assigned] (BEAM-3472) Create a callback triggered at the end of a batch in flink runner

[Aljoscha Krettek (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3472?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Aljoscha Krettek reassigned BEAM-3472:
--

Assignee: (was: Aljoscha Krettek)

> Create a callback triggered at the end of a batch in flink runner
> -
>
> Key: BEAM-3472
> URL: https://issues.apache.org/jira/browse/BEAM-3472
> Project: Beam
>  Issue Type: Improvement
>  Components: runner-flink
>Reporter: Etienne Chauchot
>Priority: Major
>
> In the future we might add new features to the runners for which we might 
> need to do some processing at the end of a batch. Currently there is not 
> unique place (a callback) to add this processing.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (BEAM-3494) Snapshot state of aggregated data of apache beam project is not maintained in flink's checkpointing

[Aljoscha Krettek (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3494?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Aljoscha Krettek reassigned BEAM-3494:
--

Assignee: (was: Aljoscha Krettek)

> Snapshot state of aggregated data of apache beam project is not maintained in 
> flink's checkpointing 
> 
>
> Key: BEAM-3494
> URL: https://issues.apache.org/jira/browse/BEAM-3494
> Project: Beam
>  Issue Type: Bug
>  Components: runner-flink
>Reporter: suganya
>Priority: Major
>
> We have a beam project which consumes events from kafka,does a groupby in a 
> time window(5 mins),after window elapses it pushes the events to downstream 
> for merge.This project is deployed using flink ,we have enabled checkpointing 
> to recover from failed state.
> (windowsize: 5mins , checkpointingInterval: 5mins,state.backend: filesystem)
> Offsets from kafka get checkpointed every 5 
> mins(checkpointingInterval).Before finishing the entire DAG(groupBy and 
> merge) , events offsets are getting checkpointed.So incase of any restart 
> from task-manager ,new task gets started from last successful checkpoint ,but 
> we could'nt able to get the aggregated snapshot data(data from groupBy task) 
> from the persisted checkpoint.
> Able to retrieve the last successful checkpointed offset from kafka ,but 
> couldnt able to get last aggregated data till checkpointing.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (BEAM-3529) Side Input callbacks should fire after a window has expired in addition to when the trigger would fire

[Aljoscha Krettek (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-3529?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16341044#comment-16341044
 ] 

Aljoscha Krettek commented on BEAM-3529:


There should probably also be {{ValidatesRunner}} tests about this behaviour to 
ensure that it is uniform across runners, right?

> Side Input callbacks should fire after a window has expired in addition to 
> when the trigger would fire
> --
>
> Key: BEAM-3529
> URL: https://issues.apache.org/jira/browse/BEAM-3529
> Project: Beam
>  Issue Type: Bug
>  Components: runner-direct
>Reporter: Thomas Groh
>Assignee: Thomas Groh
>Priority: Major
>
> Otherwise a streaming pipeline where elements trying to access a window that 
> has expired will not progress, even though we can be certain that there will 
> never be inputs in that window.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Jenkins build is still unstable: beam_PostCommit_Java_ValidatesRunner_Spark #3998

[Apache Jenkins Server]

See 

Build failed in Jenkins: beam_PerformanceTests_Python #837

[Apache Jenkins Server]

See 


--
Started by timer
[EnvInject] - Loading node environment variables.
Building remotely on beam7 (beam) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* 
 > +refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/*
 > git rev-parse origin/master^{commit} # timeout=10
Checking out Revision 87670e6f525f3a9e51f6603f072410f86be48447 (origin/master)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 87670e6f525f3a9e51f6603f072410f86be48447
Commit message: "Merge pull request #4495: Use non-deprecated version of 
Futures.transform"
 > git rev-list 87670e6f525f3a9e51f6603f072410f86be48447 # timeout=10
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
[EnvInject] - Executing scripts and injecting environment variables after the 
SCM step.
[EnvInject] - Injecting as environment variables the properties content 
SPARK_LOCAL_IP=127.0.0.1

[EnvInject] - Variables injected successfully.
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins4712018958907391642.sh
+ rm -rf PerfKitBenchmarker
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins2892737587523906551.sh
+ rm -rf .env
[beam_PerformanceTests_Python] $ /bin/bash -xe /tmp/jenkins437190026871285123.sh
+ virtualenv .env --system-site-packages
New python executable in 

Installing setuptools, pip, wheel...done.
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins5599878006945272551.sh
+ git clone https://github.com/GoogleCloudPlatform/PerfKitBenchmarker.git
Cloning into 'PerfKitBenchmarker'...
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins6604726722032393410.sh
+ .env/bin/pip install -r PerfKitBenchmarker/requirements.txt
Requirement already satisfied: absl-py in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 14))
Requirement already satisfied: jinja2>=2.7 in 
/usr/local/lib/python2.7/dist-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 15))
Requirement already satisfied: setuptools in ./.env/lib/python2.7/site-packages 
(from -r PerfKitBenchmarker/requirements.txt (line 16))
Requirement already satisfied: colorlog[windows]==2.6.0 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 17))
Requirement already satisfied: blinker>=1.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 18))
Requirement already satisfied: futures>=3.0.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 19))
Requirement already satisfied: PyYAML==3.12 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 20))
Requirement already satisfied: pint>=0.7 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 21))
Requirement already satisfied: numpy in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 22))
Requirement already satisfied: functools32 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 23))
Requirement already satisfied: contextlib2>=0.5.1 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 24))
Requirement already satisfied: pywinrm in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 25))
Requirement already satisfied: six in 
/home/jenkins/.local/lib/python2.7/site-packages (from absl-py->-r 
PerfKitBenchmarker/requirements.txt (line 14))
Requirement already satisfied: MarkupSafe>=0.23 in 
/usr/local/lib/python2.7/dist-packages (from jinja2>=2.7->-r 
PerfKitBenchmarker/requirements.txt (line 15))
Requirement already satisfied: colorama; extra == "windows" in 
/usr/lib/python2.7/dist-packages (from colorlog[windows]==2.6.0->-r 
PerfKitBenchmarker/requirements.txt (line 17))
Requirement already satisfied: xmltodict in 
/home/jenkins/.local/lib/python2.7/site-packages (from pywinrm->-r 
PerfKitBenchmarker/requirements.txt (line 25))
Requirement already satisfied: requests-ntlm>=0.3.0 in 
/home/jenkins/.local/lib/python2.7/site-packages (from pywinrm->-r 

Build failed in Jenkins: beam_PerformanceTests_TextIOIT #69

[Apache Jenkins Server]

See 


--
Started by timer
[EnvInject] - Loading node environment variables.
Building remotely on beam4 (beam) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* 
 > +refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/*
 > git rev-parse origin/master^{commit} # timeout=10
Checking out Revision 87670e6f525f3a9e51f6603f072410f86be48447 (origin/master)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 87670e6f525f3a9e51f6603f072410f86be48447
Commit message: "Merge pull request #4495: Use non-deprecated version of 
Futures.transform"
 > git rev-list 87670e6f525f3a9e51f6603f072410f86be48447 # timeout=10
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
[EnvInject] - Executing scripts and injecting environment variables after the 
SCM step.
[EnvInject] - Injecting as environment variables the properties content 
SPARK_LOCAL_IP=127.0.0.1

[EnvInject] - Variables injected successfully.
[beam_PerformanceTests_TextIOIT] $ /bin/bash -xe 
/tmp/jenkins7815540480675871912.sh
+ rm -rf PerfKitBenchmarker
[beam_PerformanceTests_TextIOIT] $ /bin/bash -xe 
/tmp/jenkins8631815251238095726.sh
+ rm -rf .env
[beam_PerformanceTests_TextIOIT] $ /bin/bash -xe 
/tmp/jenkins3629701768886629384.sh
+ virtualenv .env --system-site-packages
New python executable in .env/bin/python
Installing setuptools, pip...done.
[beam_PerformanceTests_TextIOIT] $ /bin/bash -xe 
/tmp/jenkins867591377812916747.sh
+ git clone https://github.com/GoogleCloudPlatform/PerfKitBenchmarker.git
Cloning into 'PerfKitBenchmarker'...
[beam_PerformanceTests_TextIOIT] $ /bin/bash -xe 
/tmp/jenkins786380478315322.sh
+ .env/bin/pip install -r PerfKitBenchmarker/requirements.txt
Requirement already satisfied (use --upgrade to upgrade): absl-py in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 14))
Requirement already satisfied (use --upgrade to upgrade): jinja2>=2.7 in 
/usr/local/lib/python2.7/dist-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 15))
Requirement already satisfied (use --upgrade to upgrade): setuptools in 
./.env/lib/python2.7/site-packages (from -r PerfKitBenchmarker/requirements.txt 
(line 16))
Requirement already satisfied (use --upgrade to upgrade): 
colorlog[windows]==2.6.0 in /home/jenkins/.local/lib/python2.7/site-packages 
(from -r PerfKitBenchmarker/requirements.txt (line 17))
  Installing extra requirements: 'windows'
Requirement already satisfied (use --upgrade to upgrade): blinker>=1.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 18))
Requirement already satisfied (use --upgrade to upgrade): futures>=3.0.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 19))
Requirement already satisfied (use --upgrade to upgrade): PyYAML==3.12 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 20))
Requirement already satisfied (use --upgrade to upgrade): pint>=0.7 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 21))
Requirement already satisfied (use --upgrade to upgrade): numpy in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 22))
Requirement already satisfied (use --upgrade to upgrade): functools32 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 23))
Requirement already satisfied (use --upgrade to upgrade): contextlib2>=0.5.1 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 24))
Requirement already satisfied (use --upgrade to upgrade): pywinrm in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 25))
Cleaning up...
[beam_PerformanceTests_TextIOIT] $ /bin/bash -xe 
/tmp/jenkins1238115198129649874.sh
+ .env/bin/pip install -e 'src/sdks/python/[gcp,test]'
Obtaining 
file://
  Running setup.py 
(path:
 egg_info for package from 
file://

Build failed in Jenkins: beam_PerformanceTests_Compressed_TextIOIT #61

[Apache Jenkins Server]

See 


--
Started by timer
[EnvInject] - Loading node environment variables.
Building remotely on beam6 (beam) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* 
 > +refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/*
 > git rev-parse origin/master^{commit} # timeout=10
Checking out Revision 87670e6f525f3a9e51f6603f072410f86be48447 (origin/master)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 87670e6f525f3a9e51f6603f072410f86be48447
Commit message: "Merge pull request #4495: Use non-deprecated version of 
Futures.transform"
 > git rev-list 87670e6f525f3a9e51f6603f072410f86be48447 # timeout=10
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
[EnvInject] - Executing scripts and injecting environment variables after the 
SCM step.
[EnvInject] - Injecting as environment variables the properties content 
SPARK_LOCAL_IP=127.0.0.1

[EnvInject] - Variables injected successfully.
[beam_PerformanceTests_Compressed_TextIOIT] $ /bin/bash -xe 
/tmp/jenkins2249986718433247417.sh
+ rm -rf PerfKitBenchmarker
[beam_PerformanceTests_Compressed_TextIOIT] $ /bin/bash -xe 
/tmp/jenkins7326306588681968481.sh
+ rm -rf .env
[beam_PerformanceTests_Compressed_TextIOIT] $ /bin/bash -xe 
/tmp/jenkins5087832807068688000.sh
+ virtualenv .env --system-site-packages
New python executable in .env/bin/python
Installing setuptools, pip...done.
[beam_PerformanceTests_Compressed_TextIOIT] $ /bin/bash -xe 
/tmp/jenkins4688154226428429396.sh
+ git clone https://github.com/GoogleCloudPlatform/PerfKitBenchmarker.git
Cloning into 'PerfKitBenchmarker'...
[beam_PerformanceTests_Compressed_TextIOIT] $ /bin/bash -xe 
/tmp/jenkins8721423634068608510.sh
+ .env/bin/pip install -r PerfKitBenchmarker/requirements.txt
Requirement already satisfied (use --upgrade to upgrade): absl-py in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 14))
Requirement already satisfied (use --upgrade to upgrade): jinja2>=2.7 in 
/usr/local/lib/python2.7/dist-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 15))
Requirement already satisfied (use --upgrade to upgrade): setuptools in 
./.env/lib/python2.7/site-packages (from -r PerfKitBenchmarker/requirements.txt 
(line 16))
Requirement already satisfied (use --upgrade to upgrade): 
colorlog[windows]==2.6.0 in /home/jenkins/.local/lib/python2.7/site-packages 
(from -r PerfKitBenchmarker/requirements.txt (line 17))
  Installing extra requirements: 'windows'
Requirement already satisfied (use --upgrade to upgrade): blinker>=1.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 18))
Requirement already satisfied (use --upgrade to upgrade): futures>=3.0.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 19))
Requirement already satisfied (use --upgrade to upgrade): PyYAML==3.12 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 20))
Requirement already satisfied (use --upgrade to upgrade): pint>=0.7 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 21))
Requirement already satisfied (use --upgrade to upgrade): numpy in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 22))
Requirement already satisfied (use --upgrade to upgrade): functools32 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 23))
Requirement already satisfied (use --upgrade to upgrade): contextlib2>=0.5.1 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 24))
Requirement already satisfied (use --upgrade to upgrade): pywinrm in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 25))
Cleaning up...
[beam_PerformanceTests_Compressed_TextIOIT] $ /bin/bash -xe 
/tmp/jenkins2107551679286835726.sh
+ .env/bin/pip install -e 'src/sdks/python/[gcp,test]'
Obtaining 
file://
  Running setup.py 
(path:
 egg_info for 

Build failed in Jenkins: beam_PerformanceTests_Spark #1281

[Apache Jenkins Server]

See 


--
Started by timer
[EnvInject] - Loading node environment variables.
Building remotely on beam2 (beam) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* 
 > +refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/*
 > git rev-parse origin/master^{commit} # timeout=10
Checking out Revision 87670e6f525f3a9e51f6603f072410f86be48447 (origin/master)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 87670e6f525f3a9e51f6603f072410f86be48447
Commit message: "Merge pull request #4495: Use non-deprecated version of 
Futures.transform"
 > git rev-list 87670e6f525f3a9e51f6603f072410f86be48447 # timeout=10
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
[EnvInject] - Executing scripts and injecting environment variables after the 
SCM step.
[EnvInject] - Injecting as environment variables the properties content 
SPARK_LOCAL_IP=127.0.0.1

[EnvInject] - Variables injected successfully.
[beam_PerformanceTests_Spark] $ /bin/bash -xe /tmp/jenkins5248569281124847852.sh
+ rm -rf PerfKitBenchmarker
[beam_PerformanceTests_Spark] $ /bin/bash -xe /tmp/jenkins5436234538542473692.sh
+ rm -rf .env
[beam_PerformanceTests_Spark] $ /bin/bash -xe /tmp/jenkins8663024454447526277.sh
+ virtualenv .env --system-site-packages
New python executable in 

Installing setuptools, pip, wheel...done.
[beam_PerformanceTests_Spark] $ /bin/bash -xe /tmp/jenkins716723580150727089.sh
+ git clone https://github.com/GoogleCloudPlatform/PerfKitBenchmarker.git
Cloning into 'PerfKitBenchmarker'...
[beam_PerformanceTests_Spark] $ /bin/bash -xe /tmp/jenkins8485804605467806059.sh
+ .env/bin/pip install -r PerfKitBenchmarker/requirements.txt
Requirement already satisfied: absl-py in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 14))
Requirement already satisfied: jinja2>=2.7 in 
/usr/local/lib/python2.7/dist-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 15))
Requirement already satisfied: setuptools in ./.env/lib/python2.7/site-packages 
(from -r PerfKitBenchmarker/requirements.txt (line 16))
Requirement already satisfied: colorlog[windows]==2.6.0 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 17))
Requirement already satisfied: blinker>=1.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 18))
Requirement already satisfied: futures>=3.0.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 19))
Requirement already satisfied: PyYAML==3.12 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 20))
Requirement already satisfied: pint>=0.7 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 21))
Requirement already satisfied: numpy in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 22))
Requirement already satisfied: functools32 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 23))
Requirement already satisfied: contextlib2>=0.5.1 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 24))
Requirement already satisfied: pywinrm in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 25))
Requirement already satisfied: six in 
/home/jenkins/.local/lib/python2.7/site-packages (from absl-py->-r 
PerfKitBenchmarker/requirements.txt (line 14))
Requirement already satisfied: MarkupSafe in 
/usr/local/lib/python2.7/dist-packages (from jinja2>=2.7->-r 
PerfKitBenchmarker/requirements.txt (line 15))
Requirement already satisfied: colorama; extra == "windows" in 
/usr/lib/python2.7/dist-packages (from colorlog[windows]==2.6.0->-r 
PerfKitBenchmarker/requirements.txt (line 17))
Requirement already satisfied: xmltodict in 
/home/jenkins/.local/lib/python2.7/site-packages (from pywinrm->-r 
PerfKitBenchmarker/requirements.txt (line 25))
Requirement already satisfied: requests-ntlm>=0.3.0 in 
/home/jenkins/.local/lib/python2.7/site-packages (from pywinrm->-r 
PerfKitBenchmarker/requirements.txt (line 25))

Build failed in Jenkins: beam_PerformanceTests_AvroIOIT #62

[Apache Jenkins Server]

See 


--
Started by timer
[EnvInject] - Loading node environment variables.
Building remotely on beam8 (beam) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* 
 > +refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/*
 > git rev-parse origin/master^{commit} # timeout=10
Checking out Revision 87670e6f525f3a9e51f6603f072410f86be48447 (origin/master)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 87670e6f525f3a9e51f6603f072410f86be48447
Commit message: "Merge pull request #4495: Use non-deprecated version of 
Futures.transform"
 > git rev-list 87670e6f525f3a9e51f6603f072410f86be48447 # timeout=10
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
[EnvInject] - Executing scripts and injecting environment variables after the 
SCM step.
[EnvInject] - Injecting as environment variables the properties content 
SPARK_LOCAL_IP=127.0.0.1

[EnvInject] - Variables injected successfully.
[beam_PerformanceTests_AvroIOIT] $ /bin/bash -xe 
/tmp/jenkins5304391269593045151.sh
+ rm -rf PerfKitBenchmarker
[beam_PerformanceTests_AvroIOIT] $ /bin/bash -xe 
/tmp/jenkins3412370931802405759.sh
+ rm -rf .env
[beam_PerformanceTests_AvroIOIT] $ /bin/bash -xe 
/tmp/jenkins3129571439033993674.sh
+ virtualenv .env --system-site-packages
New python executable in .env/bin/python
Installing setuptools, pip...done.
[beam_PerformanceTests_AvroIOIT] $ /bin/bash -xe 
/tmp/jenkins2584643175143244570.sh
+ git clone https://github.com/GoogleCloudPlatform/PerfKitBenchmarker.git
Cloning into 'PerfKitBenchmarker'...
[beam_PerformanceTests_AvroIOIT] $ /bin/bash -xe 
/tmp/jenkins3641051970805452060.sh
+ .env/bin/pip install -r PerfKitBenchmarker/requirements.txt
Requirement already satisfied (use --upgrade to upgrade): absl-py in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 14))
Requirement already satisfied (use --upgrade to upgrade): jinja2>=2.7 in 
/usr/local/lib/python2.7/dist-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 15))
Requirement already satisfied (use --upgrade to upgrade): setuptools in 
./.env/lib/python2.7/site-packages (from -r PerfKitBenchmarker/requirements.txt 
(line 16))
Requirement already satisfied (use --upgrade to upgrade): 
colorlog[windows]==2.6.0 in /home/jenkins/.local/lib/python2.7/site-packages 
(from -r PerfKitBenchmarker/requirements.txt (line 17))
  Installing extra requirements: 'windows'
Requirement already satisfied (use --upgrade to upgrade): blinker>=1.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 18))
Requirement already satisfied (use --upgrade to upgrade): futures>=3.0.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 19))
Requirement already satisfied (use --upgrade to upgrade): PyYAML==3.12 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 20))
Requirement already satisfied (use --upgrade to upgrade): pint>=0.7 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 21))
Requirement already satisfied (use --upgrade to upgrade): numpy in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 22))
Requirement already satisfied (use --upgrade to upgrade): functools32 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 23))
Requirement already satisfied (use --upgrade to upgrade): contextlib2>=0.5.1 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 24))
Requirement already satisfied (use --upgrade to upgrade): pywinrm in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 25))
Cleaning up...
[beam_PerformanceTests_AvroIOIT] $ /bin/bash -xe 
/tmp/jenkins8970317123305212115.sh
+ .env/bin/pip install -e 'src/sdks/python/[gcp,test]'
Obtaining 
file://
  Running setup.py 
(path:
 egg_info for package from 
file://