[jira] [Commented] (BEAM-3392) Verify default container image names in Python SDK

[JIRA]

[ 
https://issues.apache.org/jira/browse/BEAM-3392?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16340618#comment-16340618
 ] 

Jean-Baptiste Onofré commented on BEAM-3392:


Is it really a blocker for Beam 2.3.0 ? AFAIU, it seems more a Dataflow task, 
right ?

> Verify default container image names in Python SDK
> --
>
> Key: BEAM-3392
> URL: https://issues.apache.org/jira/browse/BEAM-3392
> Project: Beam
>  Issue Type: Test
>  Components: runner-dataflow
>Affects Versions: 2.3.0
>Reporter: Valentyn Tymofieiev
>Assignee: Thomas Groh
>Priority: Blocker
> Fix For: 2.3.0
>
>
> Verify naming pattern for python containers in 2.3.0 release:
> - default legacy worker harness container image: 
> dataflow.gcr.io/v1beta3/python:2.3.0
> - pinned runner harness container image: dataflow.gcr.io/v1beta3/harness:2.3.0
> - default SDK harness container name (portable pipelines): name may change, 
> exact pattern TBD.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (BEAM-3087) Extend lock scope in Flink UnboundedSourceWrapper

[JIRA]

[ 
https://issues.apache.org/jira/browse/BEAM-3087?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16340619#comment-16340619
 ] 

Jean-Baptiste Onofré commented on BEAM-3087:


[~aljoscha] any update on this one ? If we have no news soon, I will bump to 
2.4.0.

> Extend lock scope in Flink UnboundedSourceWrapper
> -
>
> Key: BEAM-3087
> URL: https://issues.apache.org/jira/browse/BEAM-3087
> Project: Beam
>  Issue Type: Bug
>  Components: runner-flink
>Reporter: Aljoscha Krettek
>Priority: Critical
> Fix For: 2.3.0
>
>
> In {{UnboundedSourceWrapper}} the lock scope is not big enough: we 
> synchronise in {{emitElement()}} but should instead synchronise inside the 
> reader loop in {{run()}} because the {{Source}} interface does not allow 
> concurrent calls.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Jenkins build is still unstable: beam_PostCommit_Java_ValidatesRunner_Spark #3997

[Apache Jenkins Server]

See 

Build failed in Jenkins: beam_PerformanceTests_Python #836

[Apache Jenkins Server]

See 


Changes:

[altay] Use non-deprecated version of Futures.transform

--
Started by timer
[EnvInject] - Loading node environment variables.
Building remotely on beam7 (beam) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* 
 > +refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/*
 > git rev-parse origin/master^{commit} # timeout=10
Checking out Revision 87670e6f525f3a9e51f6603f072410f86be48447 (origin/master)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 87670e6f525f3a9e51f6603f072410f86be48447
Commit message: "Merge pull request #4495: Use non-deprecated version of 
Futures.transform"
 > git rev-list 0f554fd51414381646c32b36f6f87a14adc0fad8 # timeout=10
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
[EnvInject] - Executing scripts and injecting environment variables after the 
SCM step.
[EnvInject] - Injecting as environment variables the properties content 
SPARK_LOCAL_IP=127.0.0.1

[EnvInject] - Variables injected successfully.
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins1998204094733082615.sh
+ rm -rf PerfKitBenchmarker
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins8067254333195029362.sh
+ rm -rf .env
[beam_PerformanceTests_Python] $ /bin/bash -xe /tmp/jenkins961330500059049560.sh
+ virtualenv .env --system-site-packages
New python executable in 

Installing setuptools, pip, wheel...done.
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins7895692462601518943.sh
+ git clone https://github.com/GoogleCloudPlatform/PerfKitBenchmarker.git
Cloning into 'PerfKitBenchmarker'...
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins9203070869092914116.sh
+ .env/bin/pip install -r PerfKitBenchmarker/requirements.txt
Requirement already satisfied: absl-py in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 14))
Requirement already satisfied: jinja2>=2.7 in 
/usr/local/lib/python2.7/dist-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 15))
Requirement already satisfied: setuptools in ./.env/lib/python2.7/site-packages 
(from -r PerfKitBenchmarker/requirements.txt (line 16))
Requirement already satisfied: colorlog[windows]==2.6.0 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 17))
Requirement already satisfied: blinker>=1.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 18))
Requirement already satisfied: futures>=3.0.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 19))
Requirement already satisfied: PyYAML==3.12 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 20))
Requirement already satisfied: pint>=0.7 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 21))
Requirement already satisfied: numpy in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 22))
Requirement already satisfied: functools32 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 23))
Requirement already satisfied: contextlib2>=0.5.1 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 24))
Requirement already satisfied: pywinrm in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 25))
Requirement already satisfied: six in 
/home/jenkins/.local/lib/python2.7/site-packages (from absl-py->-r 
PerfKitBenchmarker/requirements.txt (line 14))
Requirement already satisfied: MarkupSafe>=0.23 in 
/usr/local/lib/python2.7/dist-packages (from jinja2>=2.7->-r 
PerfKitBenchmarker/requirements.txt (line 15))
Requirement already satisfied: colorama; extra == "windows" in 
/usr/lib/python2.7/dist-packages (from colorlog[windows]==2.6.0->-r 
PerfKitBenchmarker/requirements.txt (line 17))
Requirement already satisfied: xmltodict in 
/home/jenkins/.local/lib/python2.7/site-packages (from pywinrm->-r 
PerfKitBenchmarker/requirements.txt (line 25))
Requirement already satisfied: requests-ntlm>=0.3.0 in 

Build failed in Jenkins: beam_PerformanceTests_Spark #1280

[Apache Jenkins Server]

See 


Changes:

[altay] Use non-deprecated version of Futures.transform

--
Started by timer
[EnvInject] - Loading node environment variables.
Building remotely on beam2 (beam) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* 
 > +refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/*
 > git rev-parse origin/master^{commit} # timeout=10
Checking out Revision 87670e6f525f3a9e51f6603f072410f86be48447 (origin/master)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 87670e6f525f3a9e51f6603f072410f86be48447
Commit message: "Merge pull request #4495: Use non-deprecated version of 
Futures.transform"
 > git rev-list 0f554fd51414381646c32b36f6f87a14adc0fad8 # timeout=10
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
[EnvInject] - Executing scripts and injecting environment variables after the 
SCM step.
[EnvInject] - Injecting as environment variables the properties content 
SPARK_LOCAL_IP=127.0.0.1

[EnvInject] - Variables injected successfully.
[beam_PerformanceTests_Spark] $ /bin/bash -xe /tmp/jenkins8002760368439645869.sh
+ rm -rf PerfKitBenchmarker
[beam_PerformanceTests_Spark] $ /bin/bash -xe /tmp/jenkins4079171691427536275.sh
+ rm -rf .env
[beam_PerformanceTests_Spark] $ /bin/bash -xe /tmp/jenkins240082482552373950.sh
+ virtualenv .env --system-site-packages
New python executable in 

Installing setuptools, pip, wheel...done.
[beam_PerformanceTests_Spark] $ /bin/bash -xe /tmp/jenkins5106933356521909811.sh
+ git clone https://github.com/GoogleCloudPlatform/PerfKitBenchmarker.git
Cloning into 'PerfKitBenchmarker'...
[beam_PerformanceTests_Spark] $ /bin/bash -xe /tmp/jenkins24223413178956373.sh
+ .env/bin/pip install -r PerfKitBenchmarker/requirements.txt
Requirement already satisfied: absl-py in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 14))
Requirement already satisfied: jinja2>=2.7 in 
/usr/local/lib/python2.7/dist-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 15))
Requirement already satisfied: setuptools in ./.env/lib/python2.7/site-packages 
(from -r PerfKitBenchmarker/requirements.txt (line 16))
Requirement already satisfied: colorlog[windows]==2.6.0 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 17))
Requirement already satisfied: blinker>=1.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 18))
Requirement already satisfied: futures>=3.0.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 19))
Requirement already satisfied: PyYAML==3.12 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 20))
Requirement already satisfied: pint>=0.7 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 21))
Requirement already satisfied: numpy in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 22))
Requirement already satisfied: functools32 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 23))
Requirement already satisfied: contextlib2>=0.5.1 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 24))
Requirement already satisfied: pywinrm in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 25))
Requirement already satisfied: six in 
/home/jenkins/.local/lib/python2.7/site-packages (from absl-py->-r 
PerfKitBenchmarker/requirements.txt (line 14))
Requirement already satisfied: MarkupSafe in 
/usr/local/lib/python2.7/dist-packages (from jinja2>=2.7->-r 
PerfKitBenchmarker/requirements.txt (line 15))
Requirement already satisfied: colorama; extra == "windows" in 
/usr/lib/python2.7/dist-packages (from colorlog[windows]==2.6.0->-r 
PerfKitBenchmarker/requirements.txt (line 17))
Requirement already satisfied: xmltodict in 
/home/jenkins/.local/lib/python2.7/site-packages (from pywinrm->-r 
PerfKitBenchmarker/requirements.txt (line 25))
Requirement already satisfied: requests-ntlm>=0.3.0 in 

Build failed in Jenkins: beam_PerformanceTests_AvroIOIT #61

[Apache Jenkins Server]

See 


Changes:

[altay] Use non-deprecated version of Futures.transform

--
Started by timer
[EnvInject] - Loading node environment variables.
Building remotely on beam8 (beam) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* 
 > +refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/*
 > git rev-parse origin/master^{commit} # timeout=10
Checking out Revision 87670e6f525f3a9e51f6603f072410f86be48447 (origin/master)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 87670e6f525f3a9e51f6603f072410f86be48447
Commit message: "Merge pull request #4495: Use non-deprecated version of 
Futures.transform"
 > git rev-list 0f554fd51414381646c32b36f6f87a14adc0fad8 # timeout=10
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
[EnvInject] - Executing scripts and injecting environment variables after the 
SCM step.
[EnvInject] - Injecting as environment variables the properties content 
SPARK_LOCAL_IP=127.0.0.1

[EnvInject] - Variables injected successfully.
[beam_PerformanceTests_AvroIOIT] $ /bin/bash -xe 
/tmp/jenkins6891595929917195831.sh
+ rm -rf PerfKitBenchmarker
[beam_PerformanceTests_AvroIOIT] $ /bin/bash -xe 
/tmp/jenkins1075521425140444034.sh
+ rm -rf .env
[beam_PerformanceTests_AvroIOIT] $ /bin/bash -xe 
/tmp/jenkins3713059594416054419.sh
+ virtualenv .env --system-site-packages
New python executable in .env/bin/python
Installing setuptools, pip...done.
[beam_PerformanceTests_AvroIOIT] $ /bin/bash -xe 
/tmp/jenkins6970150288771939323.sh
+ git clone https://github.com/GoogleCloudPlatform/PerfKitBenchmarker.git
Cloning into 'PerfKitBenchmarker'...
[beam_PerformanceTests_AvroIOIT] $ /bin/bash -xe 
/tmp/jenkins354755030219004224.sh
+ .env/bin/pip install -r PerfKitBenchmarker/requirements.txt
Requirement already satisfied (use --upgrade to upgrade): absl-py in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 14))
Requirement already satisfied (use --upgrade to upgrade): jinja2>=2.7 in 
/usr/local/lib/python2.7/dist-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 15))
Requirement already satisfied (use --upgrade to upgrade): setuptools in 
./.env/lib/python2.7/site-packages (from -r PerfKitBenchmarker/requirements.txt 
(line 16))
Requirement already satisfied (use --upgrade to upgrade): 
colorlog[windows]==2.6.0 in /home/jenkins/.local/lib/python2.7/site-packages 
(from -r PerfKitBenchmarker/requirements.txt (line 17))
  Installing extra requirements: 'windows'
Requirement already satisfied (use --upgrade to upgrade): blinker>=1.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 18))
Requirement already satisfied (use --upgrade to upgrade): futures>=3.0.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 19))
Requirement already satisfied (use --upgrade to upgrade): PyYAML==3.12 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 20))
Requirement already satisfied (use --upgrade to upgrade): pint>=0.7 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 21))
Requirement already satisfied (use --upgrade to upgrade): numpy in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 22))
Requirement already satisfied (use --upgrade to upgrade): functools32 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 23))
Requirement already satisfied (use --upgrade to upgrade): contextlib2>=0.5.1 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 24))
Requirement already satisfied (use --upgrade to upgrade): pywinrm in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 25))
Cleaning up...
[beam_PerformanceTests_AvroIOIT] $ /bin/bash -xe 
/tmp/jenkins8027871229984389827.sh
+ .env/bin/pip install -e 'src/sdks/python/[gcp,test]'
Obtaining 
file://
  Running setup.py 
(path:
 egg_info for package from 

Build failed in Jenkins: beam_PerformanceTests_Compressed_TextIOIT #60

[Apache Jenkins Server]

See 


Changes:

[altay] Use non-deprecated version of Futures.transform

--
Started by timer
[EnvInject] - Loading node environment variables.
Building remotely on beam6 (beam) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* 
 > +refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/*
 > git rev-parse origin/master^{commit} # timeout=10
Checking out Revision 87670e6f525f3a9e51f6603f072410f86be48447 (origin/master)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 87670e6f525f3a9e51f6603f072410f86be48447
Commit message: "Merge pull request #4495: Use non-deprecated version of 
Futures.transform"
 > git rev-list 0f554fd51414381646c32b36f6f87a14adc0fad8 # timeout=10
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
[EnvInject] - Executing scripts and injecting environment variables after the 
SCM step.
[EnvInject] - Injecting as environment variables the properties content 
SPARK_LOCAL_IP=127.0.0.1

[EnvInject] - Variables injected successfully.
[beam_PerformanceTests_Compressed_TextIOIT] $ /bin/bash -xe 
/tmp/jenkins6044480158281031814.sh
+ rm -rf PerfKitBenchmarker
[beam_PerformanceTests_Compressed_TextIOIT] $ /bin/bash -xe 
/tmp/jenkins7968179424697590037.sh
+ rm -rf .env
[beam_PerformanceTests_Compressed_TextIOIT] $ /bin/bash -xe 
/tmp/jenkins814610701827164233.sh
+ virtualenv .env --system-site-packages
New python executable in .env/bin/python
Installing setuptools, pip...done.
[beam_PerformanceTests_Compressed_TextIOIT] $ /bin/bash -xe 
/tmp/jenkins2516866847570182418.sh
+ git clone https://github.com/GoogleCloudPlatform/PerfKitBenchmarker.git
Cloning into 'PerfKitBenchmarker'...
[beam_PerformanceTests_Compressed_TextIOIT] $ /bin/bash -xe 
/tmp/jenkins728928372835749.sh
+ .env/bin/pip install -r PerfKitBenchmarker/requirements.txt
Requirement already satisfied (use --upgrade to upgrade): absl-py in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 14))
Requirement already satisfied (use --upgrade to upgrade): jinja2>=2.7 in 
/usr/local/lib/python2.7/dist-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 15))
Requirement already satisfied (use --upgrade to upgrade): setuptools in 
./.env/lib/python2.7/site-packages (from -r PerfKitBenchmarker/requirements.txt 
(line 16))
Requirement already satisfied (use --upgrade to upgrade): 
colorlog[windows]==2.6.0 in /home/jenkins/.local/lib/python2.7/site-packages 
(from -r PerfKitBenchmarker/requirements.txt (line 17))
  Installing extra requirements: 'windows'
Requirement already satisfied (use --upgrade to upgrade): blinker>=1.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 18))
Requirement already satisfied (use --upgrade to upgrade): futures>=3.0.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 19))
Requirement already satisfied (use --upgrade to upgrade): PyYAML==3.12 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 20))
Requirement already satisfied (use --upgrade to upgrade): pint>=0.7 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 21))
Requirement already satisfied (use --upgrade to upgrade): numpy in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 22))
Requirement already satisfied (use --upgrade to upgrade): functools32 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 23))
Requirement already satisfied (use --upgrade to upgrade): contextlib2>=0.5.1 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 24))
Requirement already satisfied (use --upgrade to upgrade): pywinrm in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 25))
Cleaning up...
[beam_PerformanceTests_Compressed_TextIOIT] $ /bin/bash -xe 
/tmp/jenkins6499776541467469383.sh
+ .env/bin/pip install -e 'src/sdks/python/[gcp,test]'
Obtaining 
file://
  Running setup.py 

Build failed in Jenkins: beam_PerformanceTests_TFRecordIOIT #60

[Apache Jenkins Server]

See 


Changes:

[altay] Use non-deprecated version of Futures.transform

--
Started by timer
[EnvInject] - Loading node environment variables.
Building remotely on beam4 (beam) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* 
 > +refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/*
 > git rev-parse origin/master^{commit} # timeout=10
Checking out Revision 87670e6f525f3a9e51f6603f072410f86be48447 (origin/master)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 87670e6f525f3a9e51f6603f072410f86be48447
Commit message: "Merge pull request #4495: Use non-deprecated version of 
Futures.transform"
 > git rev-list 0f554fd51414381646c32b36f6f87a14adc0fad8 # timeout=10
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
[EnvInject] - Executing scripts and injecting environment variables after the 
SCM step.
[EnvInject] - Injecting as environment variables the properties content 
SPARK_LOCAL_IP=127.0.0.1

[EnvInject] - Variables injected successfully.
[beam_PerformanceTests_TFRecordIOIT] $ /bin/bash -xe 
/tmp/jenkins7278623854359052875.sh
+ rm -rf PerfKitBenchmarker
[beam_PerformanceTests_TFRecordIOIT] $ /bin/bash -xe 
/tmp/jenkins4833243277027706449.sh
+ rm -rf .env
[beam_PerformanceTests_TFRecordIOIT] $ /bin/bash -xe 
/tmp/jenkins884936741592994093.sh
+ virtualenv .env --system-site-packages
New python executable in .env/bin/python
Installing setuptools, pip...done.
[beam_PerformanceTests_TFRecordIOIT] $ /bin/bash -xe 
/tmp/jenkins3578285256125022584.sh
+ git clone https://github.com/GoogleCloudPlatform/PerfKitBenchmarker.git
Cloning into 'PerfKitBenchmarker'...
[beam_PerformanceTests_TFRecordIOIT] $ /bin/bash -xe 
/tmp/jenkins6827446811104574980.sh
+ .env/bin/pip install -r PerfKitBenchmarker/requirements.txt
Requirement already satisfied (use --upgrade to upgrade): absl-py in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 14))
Requirement already satisfied (use --upgrade to upgrade): jinja2>=2.7 in 
/usr/local/lib/python2.7/dist-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 15))
Requirement already satisfied (use --upgrade to upgrade): setuptools in 
./.env/lib/python2.7/site-packages (from -r PerfKitBenchmarker/requirements.txt 
(line 16))
Requirement already satisfied (use --upgrade to upgrade): 
colorlog[windows]==2.6.0 in /home/jenkins/.local/lib/python2.7/site-packages 
(from -r PerfKitBenchmarker/requirements.txt (line 17))
  Installing extra requirements: 'windows'
Requirement already satisfied (use --upgrade to upgrade): blinker>=1.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 18))
Requirement already satisfied (use --upgrade to upgrade): futures>=3.0.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 19))
Requirement already satisfied (use --upgrade to upgrade): PyYAML==3.12 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 20))
Requirement already satisfied (use --upgrade to upgrade): pint>=0.7 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 21))
Requirement already satisfied (use --upgrade to upgrade): numpy in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 22))
Requirement already satisfied (use --upgrade to upgrade): functools32 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 23))
Requirement already satisfied (use --upgrade to upgrade): contextlib2>=0.5.1 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 24))
Requirement already satisfied (use --upgrade to upgrade): pywinrm in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 25))
Cleaning up...
[beam_PerformanceTests_TFRecordIOIT] $ /bin/bash -xe 
/tmp/jenkins4479272954348168160.sh
+ .env/bin/pip install -e 'src/sdks/python/[gcp,test]'
Obtaining 
file://
  Running setup.py 
(path:
 

Build failed in Jenkins: beam_PerformanceTests_TextIOIT #68

[Apache Jenkins Server]

See 


Changes:

[altay] Use non-deprecated version of Futures.transform

--
Started by timer
[EnvInject] - Loading node environment variables.
Building remotely on beam4 (beam) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* 
 > +refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/*
 > git rev-parse origin/master^{commit} # timeout=10
Checking out Revision 87670e6f525f3a9e51f6603f072410f86be48447 (origin/master)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 87670e6f525f3a9e51f6603f072410f86be48447
Commit message: "Merge pull request #4495: Use non-deprecated version of 
Futures.transform"
 > git rev-list 0f554fd51414381646c32b36f6f87a14adc0fad8 # timeout=10
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
[EnvInject] - Executing scripts and injecting environment variables after the 
SCM step.
[EnvInject] - Injecting as environment variables the properties content 
SPARK_LOCAL_IP=127.0.0.1

[EnvInject] - Variables injected successfully.
[beam_PerformanceTests_TextIOIT] $ /bin/bash -xe 
/tmp/jenkins3033400496435452181.sh
+ rm -rf PerfKitBenchmarker
[beam_PerformanceTests_TextIOIT] $ /bin/bash -xe 
/tmp/jenkins6486026696384604431.sh
+ rm -rf .env
[beam_PerformanceTests_TextIOIT] $ /bin/bash -xe 
/tmp/jenkins4934913132001105602.sh
+ virtualenv .env --system-site-packages
New python executable in .env/bin/python
Installing setuptools, pip...done.
[beam_PerformanceTests_TextIOIT] $ /bin/bash -xe 
/tmp/jenkins1901186723114040818.sh
+ git clone https://github.com/GoogleCloudPlatform/PerfKitBenchmarker.git
Cloning into 'PerfKitBenchmarker'...
[beam_PerformanceTests_TextIOIT] $ /bin/bash -xe 
/tmp/jenkins7209934064370210992.sh
+ .env/bin/pip install -r PerfKitBenchmarker/requirements.txt
Requirement already satisfied (use --upgrade to upgrade): absl-py in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 14))
Requirement already satisfied (use --upgrade to upgrade): jinja2>=2.7 in 
/usr/local/lib/python2.7/dist-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 15))
Requirement already satisfied (use --upgrade to upgrade): setuptools in 
./.env/lib/python2.7/site-packages (from -r PerfKitBenchmarker/requirements.txt 
(line 16))
Requirement already satisfied (use --upgrade to upgrade): 
colorlog[windows]==2.6.0 in /home/jenkins/.local/lib/python2.7/site-packages 
(from -r PerfKitBenchmarker/requirements.txt (line 17))
  Installing extra requirements: 'windows'
Requirement already satisfied (use --upgrade to upgrade): blinker>=1.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 18))
Requirement already satisfied (use --upgrade to upgrade): futures>=3.0.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 19))
Requirement already satisfied (use --upgrade to upgrade): PyYAML==3.12 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 20))
Requirement already satisfied (use --upgrade to upgrade): pint>=0.7 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 21))
Requirement already satisfied (use --upgrade to upgrade): numpy in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 22))
Requirement already satisfied (use --upgrade to upgrade): functools32 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 23))
Requirement already satisfied (use --upgrade to upgrade): contextlib2>=0.5.1 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 24))
Requirement already satisfied (use --upgrade to upgrade): pywinrm in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 25))
Cleaning up...
[beam_PerformanceTests_TextIOIT] $ /bin/bash -xe 
/tmp/jenkins7755192288148401508.sh
+ .env/bin/pip install -e 'src/sdks/python/[gcp,test]'
Obtaining 
file://
  Running setup.py 
(path:
 egg_info for package from 

Jenkins build is still unstable: beam_PostCommit_Java_ValidatesRunner_Spark #3996

[Apache Jenkins Server]

See 

[beam] 01/01: Merge pull request #4495: Use non-deprecated version of Futures.transform

[kenn]

This is an automated email from the ASF dual-hosted git repository.

kenn pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git

commit 87670e6f525f3a9e51f6603f072410f86be48447
Merge: 0f554fd 68944fc
Author: Kenn Knowles 
AuthorDate: Thu Jan 25 19:29:52 2018 -0800

Merge pull request #4495: Use non-deprecated version of Futures.transform

 .../org/apache/beam/runners/fnexecution/control/SdkHarnessClient.java   | 2 +-
 .../java/org/apache/beam/runners/fnexecution/data/GrpcDataService.java  | 1 +
 2 files changed, 2 insertions(+), 1 deletion(-)

-- 
To stop receiving notification emails like this one, please contact
k...@apache.org.

[beam] branch master updated (0f554fd -> 87670e6)

[kenn]

This is an automated email from the ASF dual-hosted git repository.

kenn pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from 0f554fd  Merge pull request #4417 from shashank64/master
 add 68944fc  Use non-deprecated version of Futures.transform
 new 87670e6  Merge pull request #4495: Use non-deprecated version of 
Futures.transform

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 .../org/apache/beam/runners/fnexecution/control/SdkHarnessClient.java   | 2 +-
 .../java/org/apache/beam/runners/fnexecution/data/GrpcDataService.java  | 1 +
 2 files changed, 2 insertions(+), 1 deletion(-)

-- 
To stop receiving notification emails like this one, please contact
k...@apache.org.

[beam] branch samza-runner updated (a539811 -> 0f554fd)

[kenn]

This is an automated email from the ASF dual-hosted git repository.

kenn pushed a change to branch samza-runner
in repository https://gitbox.apache.org/repos/asf/beam.git.


from a539811  Merge pull request #4451 from robertwb/fn-api-pickle-side
 add 6375069  [BEAM-2762] Python code coverage report in Postcommit
 add d41da38  fixup! Exclude auto-generated files in portability/api
 add 3ec529b  fixup! Clean up configuration and use better way to exclude 
cover in mvn build
 add a2b409c  fixup! Add more comments
 add 3f53993  fixup! Improve comments
 add 509ad27  Merge pull request #3879 from markflyhigh/coverage-report
 add c5c0cce  [BEAM-2963] Remove layer of indirection in output name 
mapping in Dataflow simplifying what needs to be passed for all portable 
pipelines to the Java SDK harness. (#4460)
 add 798da37  Add a BundleProcessor to SdkHarnessClient
 add 55844fa  Merge pull request #4266
 add 4719468  Curry CombineFn arguments into runner API protos.
 add 94d5c0b  Merge pull request #4437 from robertwb/runner-api-combine-fn
 add 3be0467  Disable combiner lifting optimization in DataflowRunner for 
fnapi
 add a69f6b6  Disable combiner lifting when only the streaming flag is set.
 add 7a4e98f  Merge pull request #4455 from aaltay/comb
 add 10a6e92  Added an example pipeline that uses stateful processing to 
output team score every time it passes a new multiple of a threshold.
 add ae1f547  Merge pull request #4272
 add b24800d  Register Environments in SdkComponents
 add a16abdf  Merge pull request #4464
 add 1057d9a  Update BUILD.md
 add 02e9a96  Merge pull request #4399 from pshevtsov/patch-1
 add 23ba6ba  NotImplementedErrror --> NotImplementedError
 add 584292e  Merge pull request #4359 from cclauss/patch-1
 add ab85e0c  Fix code comment to match code
 add 19663de  Merge pull request #4278 from itamaro/patch-1
 add 9641567  Manually specify direct runner for global-state modifying 
tests.
 add ade4fbd  FakeSource requires direct runner.
 add 194c393  Explicitly use DirectRunner in DirectRunner tests.
 add 4f92495  Document DirectRunnerOnly tests.
 add 3920cd7  Merge pull request #4454 [BEAM-3490] Explicitly use direct 
runner in non-portable tests.
 add 9b10739  Remove legacy windowfn translation.
 add d682042  Merge pull request #4212 Remove legacy windowfn translation.
 add e8a725e  Pass PipelineOptions to FileSystem constructor.
 add 6663e60  Merge pull request #4465 from udim/filesystem-options
 add f2503ba  [BEAM-3008] Adds parameters templatization for Bigtable 
(#4357)
 add 6affcb5  Migrate Flink ValidatesRunner to Gradle
 add 57e8525  Increment the Dataflow runner major version to 7.
 add 0e50e7d  Update copyright date in NOTICE
 add 62c266a  Merge pull request #4477 from yew1eb/update_copyright_date
 add 2e4b1c2  Move BeamFnDataInboundObserver to java-fn-execution
 add 7be579e  Add an implementation of the Fn API Data Service
 add f83ac0b  Merge pull request #4453
 add e70b068  [BEAM-3126] Adding a new Flatten test to Python SDK. (#4463)
 add 51ca1ee  Code compiles after auto-transition to lambda
 add 3871bbe  google-java-format
 add c824716  Removes unnecessary explicit type arguments
 add 1242a83  google-java-format
 add 10b5ad0  checkstyle fixups
 add 3d06c89  More removal of explicit type arguments
 add b0a2b0f  google-java-format
 add 1ca3a4d  Manually fixed a couple cases of bad comment formatting
 add aedb4c8  Manual fixup of some call sites where lambdas mess up coder 
inference
 add 0d87135  A couple of final example fixups
 add 0cbcf4a  Merge pull request #4470 from jkff/java8
 add cb293b9  Update printed gcloud cancel commands to include the job's 
region.
 add 12e30a3  Merge pull request #4473
 add 6389f38  [BEAM-3412] Upgrade Cloud Bigtable to 1.0.0 The 1.0.0 Cloud 
Bigtable client upgrade included a non-backwards compatible change in protobuf 
generation.  Specifically, addAll() methods no longer work correctly.  
`BigtableServiceImpl` used `addAllMutations()`, which caused run-time 
breakages.  To get around that, add each Mutation independently to the protobuf 
object. This PR also adds unit tests for BigtableServiceImpl so that read/write 
interactions do not cause runti [...]
 add 1559e37  Merge remote-tracking branch 'upstream/master' into 
upgrade_cbt_100
 add 0372db0  BigtableServiceImplTest now uses a List of ranges. The PR did 
not previously compile, since I had a pretty old version of the code in my 
development environment.  There was a non-backwards compatible change that I 
had to fix.
 add 42868b5  Updating Cloud Bigtable dependnecy to 1.0.0 in build.gradle.
 add 1095ac5  Using 1.0.0-pre3 for bigtable-proto. This should allow Cloud 
Bigtable's client to upgrade to 1.0.0 while keeping backwards compatibility 
with 

Jenkins build is back to normal : beam_PostRelease_NightlySnapshot #4

[Apache Jenkins Server]

See 

[jira] [Assigned] (BEAM-2603) Add Meter in beam metrics

[Ahmet Altay (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2603?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ahmet Altay reassigned BEAM-2603:
-

Assignee: (was: Cody)

> Add Meter in beam metrics
> -
>
> Key: BEAM-2603
> URL: https://issues.apache.org/jira/browse/BEAM-2603
> Project: Beam
>  Issue Type: Sub-task
>  Components: runner-core, sdk-java-core
>Reporter: Cody
>Priority: Major
>
> 1. Add Meter interface and implementation
> 2. Add MeterData, MeterResult. Include MeterData in metric updates, and 
> MeterResult in metric query results.
> 3. Add corresponding changes regarding MeterResult and MeterData.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: beam_PostRelease_NightlySnapshot #3

[Apache Jenkins Server]

See 


--
GitHub pull request #4494 of commit 295d7cacbb127962cdc1bb4e47f0df3ec65aa7fb, 
no merge conflicts.
Setting status of 295d7cacbb127962cdc1bb4e47f0df3ec65aa7fb to PENDING with url 
https://builds.apache.org/job/beam_PostRelease_NightlySnapshot/3/ and message: 
'Build started sha1 is merged.'
Using context: Jenkins: Google Cloud Dataflow PostRelease Tests
[EnvInject] - Loading node environment variables.
Building remotely on beam4 (beam) in workspace 

Cloning the remote Git repository
Cloning repository https://github.com/apache/beam.git
 > git init 
 >  # 
 > timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/*
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
 > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # 
 > timeout=10
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* 
 > +refs/pull/4494/*:refs/remotes/origin/pr/4494/*
 > git rev-parse refs/remotes/origin/pr/4494/merge^{commit} # timeout=10
 > git rev-parse refs/remotes/origin/origin/pr/4494/merge^{commit} # timeout=10
Checking out Revision 947c2c8fde2bc15937d692af915877462a899323 
(refs/remotes/origin/pr/4494/merge)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 947c2c8fde2bc15937d692af915877462a899323
Commit message: "Merge 295d7cacbb127962cdc1bb4e47f0df3ec65aa7fb into 
0f554fd51414381646c32b36f6f87a14adc0fad8"
First time build. Skipping changelog.
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
[EnvInject] - Executing scripts and injecting environment variables after the 
SCM step.
[EnvInject] - Injecting as environment variables the properties content 
SPARK_LOCAL_IP=127.0.0.1

[EnvInject] - Variables injected successfully.
[beam_PostRelease_NightlySnapshot] $ /bin/bash -xe 
/tmp/jenkins1397330627352195329.sh
+ cd src/release
+ . install_groovy.sh
++ set -x
++ ZIP=https://dl.bintray.com/groovy/maven/apache-groovy-binary-2.4.13.zip
++ DIR=/home/jenkins/.local/groovy
++ rm -rf /home/jenkins/.local/groovy
++ mkdir -p /home/jenkins/.local/groovy
++ pushd /home/jenkins/.local/groovy
~/.local/groovy 
~/jenkins-slave/workspace/beam_PostRelease_NightlySnapshot/src/release
++ curl -L https://dl.bintray.com/groovy/maven/apache-groovy-binary-2.4.13.zip 
--output groovy-binary.zip
  % Total% Received % Xferd  Average Speed   TimeTime Time  Current
 Dload  Upload   Total   SpentLeft  Speed
  0 00 00 0  0  0 --:--:-- --:--:-- --:--:-- 0  
0 00 00 0  0  0 --:--:-- --:--:-- --:--:-- 0
  0 00 00 0  0  0 --:--:-- --:--:-- --:--:-- 0 
25 34.8M   25 9184k0 0  6686k  0  0:00:05  0:00:01  0:00:04 9249k 
66 34.8M   66 23.2M0 0  9995k  0  0:00:03  0:00:02  0:00:01 11.6M 
92 34.8M   92 32.1M0 0  9720k  0  0:00:03  0:00:03 --:--:-- 
10.6M100 34.8M  100 34.8M0 0  9072k  0  0:00:03  0:00:03 --:--:--  
9.8M
++ unzip groovy-binary.zip
Archive:  groovy-binary.zip
   creating: groovy-2.4.13/
  inflating: groovy-2.4.13/LICENSE   
  inflating: groovy-2.4.13/NOTICE
   creating: groovy-2.4.13/lib/
  inflating: groovy-2.4.13/lib/groovy-2.4.13.jar  
  inflating: groovy-2.4.13/lib/commons-cli-1.2.jar  
  inflating: groovy-2.4.13/lib/ant-1.9.4.jar  
  inflating: groovy-2.4.13/lib/xstream-1.4.10.jar  
  inflating: groovy-2.4.13/lib/jansi-1.11.jar  
  inflating: groovy-2.4.13/lib/ivy-2.4.0.jar  
  inflating: groovy-2.4.13/lib/gpars-1.2.1.jar  
  inflating: groovy-2.4.13/lib/ant-launcher-1.9.4.jar  
  inflating: groovy-2.4.13/lib/xmlpull-1.1.3.1.jar  
  inflating: groovy-2.4.13/lib/multiverse-core-0.7.0.jar  
  inflating: groovy-2.4.13/lib/jsr166y-1.7.0.jar  
  inflating: groovy-2.4.13/lib/groovy-ant-2.4.13.jar  
  inflating: groovy-2.4.13/lib/groovy-bsf-2.4.13.jar  
  inflating: groovy-2.4.13/lib/groovy-console-2.4.13.jar  
  inflating: groovy-2.4.13/lib/groovy-docgenerator-2.4.13.jar  
  inflating: groovy-2.4.13/lib/groovy-groovydoc-2.4.13.jar  
  inflating: groovy-2.4.13/lib/groovy-groovysh-2.4.13.jar  
  inflating: groovy-2.4.13/lib/groovy-jmx-2.4.13.jar  
  inflating: groovy-2.4.13/lib/groovy-json-2.4.13.jar  
  inflating: groovy-2.4.13/lib/groovy-jsr223-2.4.13.jar  
  inflating: 

Build failed in Jenkins: beam_PostRelease_NightlySnapshot #2

[Apache Jenkins Server]

See 


--
GitHub pull request #4494 of commit 7b231fb4704a383e7124ce56df8127242cfb1805, 
no merge conflicts.
Setting status of 7b231fb4704a383e7124ce56df8127242cfb1805 to PENDING with url 
https://builds.apache.org/job/beam_PostRelease_NightlySnapshot/2/ and message: 
'Build started sha1 is merged.'
Using context: Jenkins: Google Cloud Dataflow PostRelease Tests
[EnvInject] - Loading node environment variables.
Building remotely on beam2 (beam) in workspace 

Cloning the remote Git repository
Cloning repository https://github.com/apache/beam.git
 > git init 
 >  # 
 > timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/*
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
 > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # 
 > timeout=10
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* 
 > +refs/pull/4494/*:refs/remotes/origin/pr/4494/*
 > git rev-parse refs/remotes/origin/pr/4494/merge^{commit} # timeout=10
 > git rev-parse refs/remotes/origin/origin/pr/4494/merge^{commit} # timeout=10
Checking out Revision e12888f6b4a827e8d99b58c50bddf5d8c517befe 
(refs/remotes/origin/pr/4494/merge)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f e12888f6b4a827e8d99b58c50bddf5d8c517befe
Commit message: "Merge 7b231fb4704a383e7124ce56df8127242cfb1805 into 
0f554fd51414381646c32b36f6f87a14adc0fad8"
First time build. Skipping changelog.
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
[EnvInject] - Executing scripts and injecting environment variables after the 
SCM step.
[EnvInject] - Injecting as environment variables the properties content 
SPARK_LOCAL_IP=127.0.0.1

[EnvInject] - Variables injected successfully.
[beam_PostRelease_NightlySnapshot] $ /bin/bash -xe 
/tmp/jenkins1872554019095897362.sh
+ cd src/release
+ . install_groovy.sh
++ set -x
++ ZIP=https://dl.bintray.com/groovy/maven/apache-groovy-binary-2.4.13.zip
++ DIR=/home/jenkins/.local/groovy
++ rm -rf /home/jenkins/.local/groovy
++ mkdir -p /home/jenkins/.local/groovy
++ pushd /home/jenkins/.local/groovy
~/.local/groovy 
~/jenkins-slave/workspace/beam_PostRelease_NightlySnapshot/src/release
++ curl -L https://dl.bintray.com/groovy/maven/apache-groovy-binary-2.4.13.zip 
--output groovy-binary.zip
  % Total% Received % Xferd  Average Speed   TimeTime Time  Current
 Dload  Upload   Total   SpentLeft  Speed
  0 00 00 0  0  0 --:--:-- --:--:-- --:--:-- 0  
0 00 00 0  0  0 --:--:-- --:--:-- --:--:-- 0
  5 34.8M5 1935k0 0  2370k  0  0:00:15 --:--:--  0:00:15 2370k 
59 34.8M   59 20.6M0 0  11.2M  0  0:00:03  0:00:01  0:00:02 
18.5M100 34.8M  100 34.8M0 0  13.5M  0  0:00:02  0:00:02 --:--:-- 
18.7M
++ unzip groovy-binary.zip
Archive:  groovy-binary.zip
   creating: groovy-2.4.13/
  inflating: groovy-2.4.13/LICENSE   
  inflating: groovy-2.4.13/NOTICE
   creating: groovy-2.4.13/lib/
  inflating: groovy-2.4.13/lib/groovy-2.4.13.jar  
  inflating: groovy-2.4.13/lib/commons-cli-1.2.jar  
  inflating: groovy-2.4.13/lib/ant-1.9.4.jar  
  inflating: groovy-2.4.13/lib/xstream-1.4.10.jar  
  inflating: groovy-2.4.13/lib/jansi-1.11.jar  
  inflating: groovy-2.4.13/lib/ivy-2.4.0.jar  
  inflating: groovy-2.4.13/lib/gpars-1.2.1.jar  
  inflating: groovy-2.4.13/lib/ant-launcher-1.9.4.jar  
  inflating: groovy-2.4.13/lib/xmlpull-1.1.3.1.jar  
  inflating: groovy-2.4.13/lib/multiverse-core-0.7.0.jar  
  inflating: groovy-2.4.13/lib/jsr166y-1.7.0.jar  
  inflating: groovy-2.4.13/lib/groovy-ant-2.4.13.jar  
  inflating: groovy-2.4.13/lib/groovy-bsf-2.4.13.jar  
  inflating: groovy-2.4.13/lib/groovy-console-2.4.13.jar  
  inflating: groovy-2.4.13/lib/groovy-docgenerator-2.4.13.jar  
  inflating: groovy-2.4.13/lib/groovy-groovydoc-2.4.13.jar  
  inflating: groovy-2.4.13/lib/groovy-groovysh-2.4.13.jar  
  inflating: groovy-2.4.13/lib/groovy-jmx-2.4.13.jar  
  inflating: groovy-2.4.13/lib/groovy-json-2.4.13.jar  
  inflating: groovy-2.4.13/lib/groovy-jsr223-2.4.13.jar  
  inflating: groovy-2.4.13/lib/groovy-nio-2.4.13.jar  
  inflating: groovy-2.4.13/lib/groovy-servlet-2.4.13.jar  
  inflating: groovy-2.4.13/lib/groovy-sql-2.4.13.jar  
  inflating: 

Jenkins build is still unstable: beam_PostCommit_Java_ValidatesRunner_Spark #3995

[Apache Jenkins Server]

See 

Build failed in Jenkins: beam_PerformanceTests_Python #835

[Apache Jenkins Server]

See 


Changes:

[shashank] fix serialization error in BigQueryIO's DynamicDestinations

[shashank] correct side input check in BigQueryIO DynamicDestination for pull

[coheigea] BEAM-3533 - Replace hard-coded UTF-8 Strings

[jbonofre] [BEAM-3466] Remove JDK 7 references in Jenkins

--
Started by timer
[EnvInject] - Loading node environment variables.
Building remotely on beam3 (beam) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* 
 > +refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/*
 > git rev-parse origin/master^{commit} # timeout=10
Checking out Revision 0f554fd51414381646c32b36f6f87a14adc0fad8 (origin/master)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 0f554fd51414381646c32b36f6f87a14adc0fad8
Commit message: "Merge pull request #4417 from shashank64/master"
 > git rev-list c07a43ee2d4d81de15d78f8ec5a52816248d1afe # timeout=10
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
[EnvInject] - Executing scripts and injecting environment variables after the 
SCM step.
[EnvInject] - Injecting as environment variables the properties content 
SPARK_LOCAL_IP=127.0.0.1

[EnvInject] - Variables injected successfully.
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins8491686870287645687.sh
+ rm -rf PerfKitBenchmarker
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins3740577391775067095.sh
+ rm -rf .env
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins3492718838635065982.sh
+ virtualenv .env --system-site-packages
New python executable in .env/bin/python
Installing setuptools, pip...done.
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins4433122496208090018.sh
+ git clone https://github.com/GoogleCloudPlatform/PerfKitBenchmarker.git
Cloning into 'PerfKitBenchmarker'...
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins6090587241159871283.sh
+ .env/bin/pip install -r PerfKitBenchmarker/requirements.txt
Requirement already satisfied (use --upgrade to upgrade): absl-py in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 14))
Requirement already satisfied (use --upgrade to upgrade): jinja2>=2.7 in 
/usr/local/lib/python2.7/dist-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 15))
Requirement already satisfied (use --upgrade to upgrade): setuptools in 
./.env/lib/python2.7/site-packages (from -r PerfKitBenchmarker/requirements.txt 
(line 16))
Requirement already satisfied (use --upgrade to upgrade): 
colorlog[windows]==2.6.0 in /home/jenkins/.local/lib/python2.7/site-packages 
(from -r PerfKitBenchmarker/requirements.txt (line 17))
  Installing extra requirements: 'windows'
Requirement already satisfied (use --upgrade to upgrade): blinker>=1.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 18))
Requirement already satisfied (use --upgrade to upgrade): futures>=3.0.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 19))
Requirement already satisfied (use --upgrade to upgrade): PyYAML==3.12 in 
/usr/local/lib/python2.7/dist-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 20))
Requirement already satisfied (use --upgrade to upgrade): pint>=0.7 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 21))
Requirement already satisfied (use --upgrade to upgrade): numpy in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 22))
Requirement already satisfied (use --upgrade to upgrade): functools32 in 
/usr/local/lib/python2.7/dist-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 23))
Requirement already satisfied (use --upgrade to upgrade): contextlib2>=0.5.1 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 24))
Requirement already satisfied (use --upgrade to upgrade): pywinrm in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 25))
Cleaning up...
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins3553162801093382282.sh
+ .env/bin/pip install -e 'src/sdks/python/[gcp,test]'
Obtaining 
file://
  Running setup.py 

Build failed in Jenkins: beam_PerformanceTests_Spark #1279

[Apache Jenkins Server]

See 


Changes:

[shashank] fix serialization error in BigQueryIO's DynamicDestinations

[shashank] correct side input check in BigQueryIO DynamicDestination for pull

[coheigea] BEAM-3533 - Replace hard-coded UTF-8 Strings

[jbonofre] [BEAM-3466] Remove JDK 7 references in Jenkins

--
Started by timer
[EnvInject] - Loading node environment variables.
Building remotely on beam4 (beam) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* 
 > +refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/*
 > git rev-parse origin/master^{commit} # timeout=10
Checking out Revision 0f554fd51414381646c32b36f6f87a14adc0fad8 (origin/master)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 0f554fd51414381646c32b36f6f87a14adc0fad8
Commit message: "Merge pull request #4417 from shashank64/master"
 > git rev-list c07a43ee2d4d81de15d78f8ec5a52816248d1afe # timeout=10
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
[EnvInject] - Executing scripts and injecting environment variables after the 
SCM step.
[EnvInject] - Injecting as environment variables the properties content 
SPARK_LOCAL_IP=127.0.0.1

[EnvInject] - Variables injected successfully.
[beam_PerformanceTests_Spark] $ /bin/bash -xe /tmp/jenkins4446360943615021798.sh
+ rm -rf PerfKitBenchmarker
[beam_PerformanceTests_Spark] $ /bin/bash -xe /tmp/jenkins6062055313325768742.sh
+ rm -rf .env
[beam_PerformanceTests_Spark] $ /bin/bash -xe /tmp/jenkins8303885279376286113.sh
+ virtualenv .env --system-site-packages
New python executable in .env/bin/python
Installing setuptools, pip...done.
[beam_PerformanceTests_Spark] $ /bin/bash -xe /tmp/jenkins9150935709800420148.sh
+ git clone https://github.com/GoogleCloudPlatform/PerfKitBenchmarker.git
Cloning into 'PerfKitBenchmarker'...
[beam_PerformanceTests_Spark] $ /bin/bash -xe /tmp/jenkins5791825342469178143.sh
+ .env/bin/pip install -r PerfKitBenchmarker/requirements.txt
Requirement already satisfied (use --upgrade to upgrade): absl-py in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 14))
Requirement already satisfied (use --upgrade to upgrade): jinja2>=2.7 in 
/usr/local/lib/python2.7/dist-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 15))
Requirement already satisfied (use --upgrade to upgrade): setuptools in 
./.env/lib/python2.7/site-packages (from -r PerfKitBenchmarker/requirements.txt 
(line 16))
Requirement already satisfied (use --upgrade to upgrade): 
colorlog[windows]==2.6.0 in /home/jenkins/.local/lib/python2.7/site-packages 
(from -r PerfKitBenchmarker/requirements.txt (line 17))
  Installing extra requirements: 'windows'
Requirement already satisfied (use --upgrade to upgrade): blinker>=1.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 18))
Requirement already satisfied (use --upgrade to upgrade): futures>=3.0.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 19))
Requirement already satisfied (use --upgrade to upgrade): PyYAML==3.12 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 20))
Requirement already satisfied (use --upgrade to upgrade): pint>=0.7 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 21))
Requirement already satisfied (use --upgrade to upgrade): numpy in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 22))
Requirement already satisfied (use --upgrade to upgrade): functools32 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 23))
Requirement already satisfied (use --upgrade to upgrade): contextlib2>=0.5.1 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 24))
Requirement already satisfied (use --upgrade to upgrade): pywinrm in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 25))
Cleaning up...
[beam_PerformanceTests_Spark] $ /bin/bash -xe /tmp/jenkins2132276002619799502.sh
+ .env/bin/pip install -e 'src/sdks/python/[gcp,test]'
Obtaining 
file://
  Running 

Build failed in Jenkins: beam_PerformanceTests_TFRecordIOIT #59

[Apache Jenkins Server]

See 


Changes:

[shashank] fix serialization error in BigQueryIO's DynamicDestinations

[shashank] correct side input check in BigQueryIO DynamicDestination for pull

[coheigea] BEAM-3533 - Replace hard-coded UTF-8 Strings

[jbonofre] [BEAM-3466] Remove JDK 7 references in Jenkins

--
Started by timer
[EnvInject] - Loading node environment variables.
Building remotely on beam4 (beam) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* 
 > +refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/*
 > git rev-parse origin/master^{commit} # timeout=10
Checking out Revision 0f554fd51414381646c32b36f6f87a14adc0fad8 (origin/master)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 0f554fd51414381646c32b36f6f87a14adc0fad8
Commit message: "Merge pull request #4417 from shashank64/master"
 > git rev-list c07a43ee2d4d81de15d78f8ec5a52816248d1afe # timeout=10
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
[EnvInject] - Executing scripts and injecting environment variables after the 
SCM step.
[EnvInject] - Injecting as environment variables the properties content 
SPARK_LOCAL_IP=127.0.0.1

[EnvInject] - Variables injected successfully.
[beam_PerformanceTests_TFRecordIOIT] $ /bin/bash -xe 
/tmp/jenkins1641288567050315292.sh
+ rm -rf PerfKitBenchmarker
[beam_PerformanceTests_TFRecordIOIT] $ /bin/bash -xe 
/tmp/jenkins5451905937393583674.sh
+ rm -rf .env
[beam_PerformanceTests_TFRecordIOIT] $ /bin/bash -xe 
/tmp/jenkins5714508279768743373.sh
+ virtualenv .env --system-site-packages
New python executable in .env/bin/python
Installing setuptools, pip...done.
[beam_PerformanceTests_TFRecordIOIT] $ /bin/bash -xe 
/tmp/jenkins4383822252317357975.sh
+ git clone https://github.com/GoogleCloudPlatform/PerfKitBenchmarker.git
Cloning into 'PerfKitBenchmarker'...
[beam_PerformanceTests_TFRecordIOIT] $ /bin/bash -xe 
/tmp/jenkins8807089809483818689.sh
+ .env/bin/pip install -r PerfKitBenchmarker/requirements.txt
Requirement already satisfied (use --upgrade to upgrade): absl-py in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 14))
Requirement already satisfied (use --upgrade to upgrade): jinja2>=2.7 in 
/usr/local/lib/python2.7/dist-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 15))
Requirement already satisfied (use --upgrade to upgrade): setuptools in 
./.env/lib/python2.7/site-packages (from -r PerfKitBenchmarker/requirements.txt 
(line 16))
Requirement already satisfied (use --upgrade to upgrade): 
colorlog[windows]==2.6.0 in /home/jenkins/.local/lib/python2.7/site-packages 
(from -r PerfKitBenchmarker/requirements.txt (line 17))
  Installing extra requirements: 'windows'
Requirement already satisfied (use --upgrade to upgrade): blinker>=1.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 18))
Requirement already satisfied (use --upgrade to upgrade): futures>=3.0.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 19))
Requirement already satisfied (use --upgrade to upgrade): PyYAML==3.12 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 20))
Requirement already satisfied (use --upgrade to upgrade): pint>=0.7 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 21))
Requirement already satisfied (use --upgrade to upgrade): numpy in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 22))
Requirement already satisfied (use --upgrade to upgrade): functools32 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 23))
Requirement already satisfied (use --upgrade to upgrade): contextlib2>=0.5.1 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 24))
Requirement already satisfied (use --upgrade to upgrade): pywinrm in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 25))
Cleaning up...
[beam_PerformanceTests_TFRecordIOIT] $ /bin/bash -xe 
/tmp/jenkins6630279250226769245.sh
+ .env/bin/pip install -e 'src/sdks/python/[gcp,test]'
Obtaining 

Build failed in Jenkins: beam_PerformanceTests_TextIOIT #67

[Apache Jenkins Server]

See 


Changes:

[shashank] fix serialization error in BigQueryIO's DynamicDestinations

[shashank] correct side input check in BigQueryIO DynamicDestination for pull

[coheigea] BEAM-3533 - Replace hard-coded UTF-8 Strings

[jbonofre] [BEAM-3466] Remove JDK 7 references in Jenkins

--
Started by timer
[EnvInject] - Loading node environment variables.
Building remotely on beam4 (beam) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* 
 > +refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/*
 > git rev-parse origin/master^{commit} # timeout=10
Checking out Revision 0f554fd51414381646c32b36f6f87a14adc0fad8 (origin/master)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 0f554fd51414381646c32b36f6f87a14adc0fad8
Commit message: "Merge pull request #4417 from shashank64/master"
 > git rev-list c07a43ee2d4d81de15d78f8ec5a52816248d1afe # timeout=10
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
[EnvInject] - Executing scripts and injecting environment variables after the 
SCM step.
[EnvInject] - Injecting as environment variables the properties content 
SPARK_LOCAL_IP=127.0.0.1

[EnvInject] - Variables injected successfully.
[beam_PerformanceTests_TextIOIT] $ /bin/bash -xe 
/tmp/jenkins8063028415580261990.sh
+ rm -rf PerfKitBenchmarker
[beam_PerformanceTests_TextIOIT] $ /bin/bash -xe 
/tmp/jenkins5123193678645243453.sh
+ rm -rf .env
[beam_PerformanceTests_TextIOIT] $ /bin/bash -xe 
/tmp/jenkins5404143407514358059.sh
+ virtualenv .env --system-site-packages
New python executable in .env/bin/python
Installing setuptools, pip...done.
[beam_PerformanceTests_TextIOIT] $ /bin/bash -xe 
/tmp/jenkins7926119406341191701.sh
+ git clone https://github.com/GoogleCloudPlatform/PerfKitBenchmarker.git
Cloning into 'PerfKitBenchmarker'...
[beam_PerformanceTests_TextIOIT] $ /bin/bash -xe 
/tmp/jenkins8404321225845644373.sh
+ .env/bin/pip install -r PerfKitBenchmarker/requirements.txt
Requirement already satisfied (use --upgrade to upgrade): absl-py in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 14))
Requirement already satisfied (use --upgrade to upgrade): jinja2>=2.7 in 
/usr/local/lib/python2.7/dist-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 15))
Requirement already satisfied (use --upgrade to upgrade): setuptools in 
./.env/lib/python2.7/site-packages (from -r PerfKitBenchmarker/requirements.txt 
(line 16))
Requirement already satisfied (use --upgrade to upgrade): 
colorlog[windows]==2.6.0 in /home/jenkins/.local/lib/python2.7/site-packages 
(from -r PerfKitBenchmarker/requirements.txt (line 17))
  Installing extra requirements: 'windows'
Requirement already satisfied (use --upgrade to upgrade): blinker>=1.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 18))
Requirement already satisfied (use --upgrade to upgrade): futures>=3.0.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 19))
Requirement already satisfied (use --upgrade to upgrade): PyYAML==3.12 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 20))
Requirement already satisfied (use --upgrade to upgrade): pint>=0.7 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 21))
Requirement already satisfied (use --upgrade to upgrade): numpy in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 22))
Requirement already satisfied (use --upgrade to upgrade): functools32 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 23))
Requirement already satisfied (use --upgrade to upgrade): contextlib2>=0.5.1 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 24))
Requirement already satisfied (use --upgrade to upgrade): pywinrm in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 25))
Cleaning up...
[beam_PerformanceTests_TextIOIT] $ /bin/bash -xe 
/tmp/jenkins6047135697427712159.sh
+ .env/bin/pip install -e 'src/sdks/python/[gcp,test]'
Obtaining 

Build failed in Jenkins: beam_PerformanceTests_Compressed_TextIOIT #59

[Apache Jenkins Server]

See 


Changes:

[shashank] fix serialization error in BigQueryIO's DynamicDestinations

[shashank] correct side input check in BigQueryIO DynamicDestination for pull

[coheigea] BEAM-3533 - Replace hard-coded UTF-8 Strings

[jbonofre] [BEAM-3466] Remove JDK 7 references in Jenkins

--
Started by timer
[EnvInject] - Loading node environment variables.
Building remotely on beam4 (beam) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* 
 > +refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/*
 > git rev-parse origin/master^{commit} # timeout=10
Checking out Revision 0f554fd51414381646c32b36f6f87a14adc0fad8 (origin/master)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 0f554fd51414381646c32b36f6f87a14adc0fad8
Commit message: "Merge pull request #4417 from shashank64/master"
 > git rev-list c07a43ee2d4d81de15d78f8ec5a52816248d1afe # timeout=10
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
[EnvInject] - Executing scripts and injecting environment variables after the 
SCM step.
[EnvInject] - Injecting as environment variables the properties content 
SPARK_LOCAL_IP=127.0.0.1

[EnvInject] - Variables injected successfully.
[beam_PerformanceTests_Compressed_TextIOIT] $ /bin/bash -xe 
/tmp/jenkins4301797212616673983.sh
+ rm -rf PerfKitBenchmarker
[beam_PerformanceTests_Compressed_TextIOIT] $ /bin/bash -xe 
/tmp/jenkins8025746728149437773.sh
+ rm -rf .env
[beam_PerformanceTests_Compressed_TextIOIT] $ /bin/bash -xe 
/tmp/jenkins2164455956134243553.sh
+ virtualenv .env --system-site-packages
New python executable in .env/bin/python
Installing setuptools, pip...done.
[beam_PerformanceTests_Compressed_TextIOIT] $ /bin/bash -xe 
/tmp/jenkins5014093506780795799.sh
+ git clone https://github.com/GoogleCloudPlatform/PerfKitBenchmarker.git
Cloning into 'PerfKitBenchmarker'...
[beam_PerformanceTests_Compressed_TextIOIT] $ /bin/bash -xe 
/tmp/jenkins7317894172099728561.sh
+ .env/bin/pip install -r PerfKitBenchmarker/requirements.txt
Requirement already satisfied (use --upgrade to upgrade): absl-py in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 14))
Requirement already satisfied (use --upgrade to upgrade): jinja2>=2.7 in 
/usr/local/lib/python2.7/dist-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 15))
Requirement already satisfied (use --upgrade to upgrade): setuptools in 
./.env/lib/python2.7/site-packages (from -r PerfKitBenchmarker/requirements.txt 
(line 16))
Requirement already satisfied (use --upgrade to upgrade): 
colorlog[windows]==2.6.0 in /home/jenkins/.local/lib/python2.7/site-packages 
(from -r PerfKitBenchmarker/requirements.txt (line 17))
  Installing extra requirements: 'windows'
Requirement already satisfied (use --upgrade to upgrade): blinker>=1.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 18))
Requirement already satisfied (use --upgrade to upgrade): futures>=3.0.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 19))
Requirement already satisfied (use --upgrade to upgrade): PyYAML==3.12 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 20))
Requirement already satisfied (use --upgrade to upgrade): pint>=0.7 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 21))
Requirement already satisfied (use --upgrade to upgrade): numpy in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 22))
Requirement already satisfied (use --upgrade to upgrade): functools32 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 23))
Requirement already satisfied (use --upgrade to upgrade): contextlib2>=0.5.1 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 24))
Requirement already satisfied (use --upgrade to upgrade): pywinrm in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 25))
Cleaning up...
[beam_PerformanceTests_Compressed_TextIOIT] $ /bin/bash -xe 
/tmp/jenkins7328593728908046511.sh
+ .env/bin/pip install -e 

Build failed in Jenkins: beam_PerformanceTests_AvroIOIT #60

[Apache Jenkins Server]

See 


Changes:

[shashank] fix serialization error in BigQueryIO's DynamicDestinations

[shashank] correct side input check in BigQueryIO DynamicDestination for pull

[coheigea] BEAM-3533 - Replace hard-coded UTF-8 Strings

[jbonofre] [BEAM-3466] Remove JDK 7 references in Jenkins

--
Started by timer
[EnvInject] - Loading node environment variables.
Building remotely on beam4 (beam) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* 
 > +refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/*
 > git rev-parse origin/master^{commit} # timeout=10
Checking out Revision 0f554fd51414381646c32b36f6f87a14adc0fad8 (origin/master)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 0f554fd51414381646c32b36f6f87a14adc0fad8
Commit message: "Merge pull request #4417 from shashank64/master"
 > git rev-list c07a43ee2d4d81de15d78f8ec5a52816248d1afe # timeout=10
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
[EnvInject] - Executing scripts and injecting environment variables after the 
SCM step.
[EnvInject] - Injecting as environment variables the properties content 
SPARK_LOCAL_IP=127.0.0.1

[EnvInject] - Variables injected successfully.
[beam_PerformanceTests_AvroIOIT] $ /bin/bash -xe 
/tmp/jenkins4554714707298800771.sh
+ rm -rf PerfKitBenchmarker
[beam_PerformanceTests_AvroIOIT] $ /bin/bash -xe 
/tmp/jenkins7243998696868474785.sh
+ rm -rf .env
[beam_PerformanceTests_AvroIOIT] $ /bin/bash -xe 
/tmp/jenkins5195414680525887899.sh
+ virtualenv .env --system-site-packages
New python executable in .env/bin/python
Installing setuptools, pip...done.
[beam_PerformanceTests_AvroIOIT] $ /bin/bash -xe 
/tmp/jenkins6708802127740400765.sh
+ git clone https://github.com/GoogleCloudPlatform/PerfKitBenchmarker.git
Cloning into 'PerfKitBenchmarker'...
[beam_PerformanceTests_AvroIOIT] $ /bin/bash -xe 
/tmp/jenkins5247087468488821597.sh
+ .env/bin/pip install -r PerfKitBenchmarker/requirements.txt
Requirement already satisfied (use --upgrade to upgrade): absl-py in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 14))
Requirement already satisfied (use --upgrade to upgrade): jinja2>=2.7 in 
/usr/local/lib/python2.7/dist-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 15))
Requirement already satisfied (use --upgrade to upgrade): setuptools in 
./.env/lib/python2.7/site-packages (from -r PerfKitBenchmarker/requirements.txt 
(line 16))
Requirement already satisfied (use --upgrade to upgrade): 
colorlog[windows]==2.6.0 in /home/jenkins/.local/lib/python2.7/site-packages 
(from -r PerfKitBenchmarker/requirements.txt (line 17))
  Installing extra requirements: 'windows'
Requirement already satisfied (use --upgrade to upgrade): blinker>=1.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 18))
Requirement already satisfied (use --upgrade to upgrade): futures>=3.0.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 19))
Requirement already satisfied (use --upgrade to upgrade): PyYAML==3.12 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 20))
Requirement already satisfied (use --upgrade to upgrade): pint>=0.7 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 21))
Requirement already satisfied (use --upgrade to upgrade): numpy in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 22))
Requirement already satisfied (use --upgrade to upgrade): functools32 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 23))
Requirement already satisfied (use --upgrade to upgrade): contextlib2>=0.5.1 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 24))
Requirement already satisfied (use --upgrade to upgrade): pywinrm in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 25))
Cleaning up...
[beam_PerformanceTests_AvroIOIT] $ /bin/bash -xe 
/tmp/jenkins6292671895881438247.sh
+ .env/bin/pip install -e 'src/sdks/python/[gcp,test]'
Obtaining 

Jenkins build is back to stable : beam_PostCommit_Java_MavenInstall #5760

[Apache Jenkins Server]

See 

[jira] [Created] (BEAM-3537) Remove DirectRunner-specific internal PValue cache, allow more general eager in-process pipeline execution

[Charles Chen (JIRA)]

Charles Chen created BEAM-3537:
--

 Summary: Remove DirectRunner-specific internal PValue cache, allow 
more general eager in-process pipeline execution
 Key: BEAM-3537
 URL: https://issues.apache.org/jira/browse/BEAM-3537
 Project: Beam
  Issue Type: Bug
  Components: sdk-py-core
Affects Versions: 2.2.0
Reporter: Charles Chen
Assignee: Charles Chen


Currently, the Python SDK supports an eager execution mode.  For example, a 
list can be directly passed into a PTransform to obtain its result:

{{result = [1, 2, 3] | MyPTransform()}}

To support this use, the Python DirectRunner has an option to cache its 
intermediate results into a PValueCache.  The above line, when run, implicitly 
creates an ephemeral pipeline and runs it with the DirectRunner.  This, 
however, adds a lot of complexity to the DirectRunner, and is not generalizable 
to other in-process Python runners (like the in-process Python FnApiRunner).

To improve this, we should remove this DirectRunner-specific implementation and 
add functionality that allows all in-process Python runners to be run in eager 
mode.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Jenkins build is still unstable: beam_PostCommit_Java_ValidatesRunner_Spark #3994

[Apache Jenkins Server]

See 

Build failed in Jenkins: beam_PostCommit_Python_ValidatesContainer_Dataflow #12

[Apache Jenkins Server]

See 


Changes:

[shashank] fix serialization error in BigQueryIO's DynamicDestinations

[shashank] correct side input check in BigQueryIO DynamicDestination for pull

--
Started by GitHub push by jkff
[EnvInject] - Loading node environment variables.
Building remotely on beam1 (beam) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* 
 > +refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/*
 > git rev-parse origin/master^{commit} # timeout=10
Checking out Revision 0f554fd51414381646c32b36f6f87a14adc0fad8 (origin/master)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 0f554fd51414381646c32b36f6f87a14adc0fad8
Commit message: "Merge pull request #4417 from shashank64/master"
 > git rev-list 5ee938ec37f729f5fca96e2b02b034505fc86fb0 # timeout=10
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
[EnvInject] - Executing scripts and injecting environment variables after the 
SCM step.
[EnvInject] - Injecting as environment variables the properties content 
SPARK_LOCAL_IP=127.0.0.1

[EnvInject] - Variables injected successfully.
[beam_PostCommit_Python_ValidatesContainer_Dataflow] $ /bin/bash -xe 
/tmp/jenkins6683258434702946390.sh
+ cd src
+ bash sdks/python/container/run_validatescontainer.sh
bash: sdks/python/container/run_validatescontainer.sh: No such file or directory
Build step 'Execute shell' marked build as failure
Not sending mail to unregistered user rob...@frantil.com

[jira] [Updated] (BEAM-2776) TextIO should support reading header lines

[Eugene Kirpichov (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2776?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Eugene Kirpichov updated BEAM-2776:
---
Priority: Minor  (was: Major)

> TextIO should support reading header lines
> --
>
> Key: BEAM-2776
> URL: https://issues.apache.org/jira/browse/BEAM-2776
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core, sdk-py-core
>Reporter: Eugene Kirpichov
>Priority: Minor
>
> Users frequently request the ability to skip some header rows when reading 
> text files.
> https://stackoverflow.com/questions/28450554/skipping-header-rows-is-it-possible-with-cloud-dataflow
> https://stackoverflow.com/questions/43551876/how-do-i-read-and-transform-csv-headers-before-bigqueryio-write
> https://stackoverflow.com/questions/41297704/reading-csv-header-with-dataflow
> https://stackoverflow.com/questions/45554466/google-cloud-dataflow-apache-beam-how-to-process-gzipped-csv-files-with-a-he
> https://stackoverflow.com/questions/44045744/how-do-i-skip-header-files-when-reading-from-google-cloud-storage-in-a-dataflow
> This is also relevant for reading file formats such as VCF, see thread 
> https://lists.apache.org/thread.html/dc7e5c3ff20d9270f06c1a298ad949da018a83f900b22d58f6b4c468@%3Cdev.beam.apache.org%3E
> Python supports this partially https://github.com/apache/beam/pull/1771/files 
> via skip_header_lines, but the header lines can have useful content, and the 
> number of header lines is not fixed (in VCF).
> We should figure out a good API for this and support this natively in TextIO. 
> The API decisions would be:
> - How do we specify how much of the beginning of each file is the header: 
> options could be e.g. a certain number of lines; or lines that start with a 
> certain character; or a custom predicate.
> - How do we make the header contents accessible to a user of TextIO. Since 
> the header can be different in each file, we can't return it as a 
> PCollectionView. Instead I suppose, when you use a header, 
> you'd need to specify a SerializableFunction, T> or 
> something like that for parsing (header, line) -> user type. Note that 
> currently TextIO.Read does not support returning a user type anyway, so 
> that'd need to be done too.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (BEAM-2776) TextIO should support reading header lines

[Eugene Kirpichov (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2776?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16340207#comment-16340207
 ] 

Eugene Kirpichov commented on BEAM-2776:


Reducing priority: This is easy to do manually using FileIO.match() + 
readMatches() and possibly doesn't warrant changes to TextIO, unless someone 
has a compelling argument to the contrary.

> TextIO should support reading header lines
> --
>
> Key: BEAM-2776
> URL: https://issues.apache.org/jira/browse/BEAM-2776
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core, sdk-py-core
>Reporter: Eugene Kirpichov
>Priority: Minor
>
> Users frequently request the ability to skip some header rows when reading 
> text files.
> https://stackoverflow.com/questions/28450554/skipping-header-rows-is-it-possible-with-cloud-dataflow
> https://stackoverflow.com/questions/43551876/how-do-i-read-and-transform-csv-headers-before-bigqueryio-write
> https://stackoverflow.com/questions/41297704/reading-csv-header-with-dataflow
> https://stackoverflow.com/questions/45554466/google-cloud-dataflow-apache-beam-how-to-process-gzipped-csv-files-with-a-he
> https://stackoverflow.com/questions/44045744/how-do-i-skip-header-files-when-reading-from-google-cloud-storage-in-a-dataflow
> This is also relevant for reading file formats such as VCF, see thread 
> https://lists.apache.org/thread.html/dc7e5c3ff20d9270f06c1a298ad949da018a83f900b22d58f6b4c468@%3Cdev.beam.apache.org%3E
> Python supports this partially https://github.com/apache/beam/pull/1771/files 
> via skip_header_lines, but the header lines can have useful content, and the 
> number of header lines is not fixed (in VCF).
> We should figure out a good API for this and support this natively in TextIO. 
> The API decisions would be:
> - How do we specify how much of the beginning of each file is the header: 
> options could be e.g. a certain number of lines; or lines that start with a 
> certain character; or a custom predicate.
> - How do we make the header contents accessible to a user of TextIO. Since 
> the header can be different in each file, we can't return it as a 
> PCollectionView. Instead I suppose, when you use a header, 
> you'd need to specify a SerializableFunction, T> or 
> something like that for parsing (header, line) -> user type. Note that 
> currently TextIO.Read does not support returning a user type anyway, so 
> that'd need to be done too.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Closed] (BEAM-2768) Fix bigquery.WriteTables generating non-unique job identifiers

[Eugene Kirpichov (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2768?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Eugene Kirpichov closed BEAM-2768.
--
   Resolution: Cannot Reproduce
Fix Version/s: Not applicable

Closing due to lack of response from reporter. Please reopen with more 
information if this reoccurs.

> Fix bigquery.WriteTables generating non-unique job identifiers
> --
>
> Key: BEAM-2768
> URL: https://issues.apache.org/jira/browse/BEAM-2768
> Project: Beam
>  Issue Type: Bug
>  Components: beam-model
>Affects Versions: 2.0.0
>Reporter: Matti Remes
>Assignee: Reuven Lax
>Priority: Major
> Fix For: Not applicable
>
>
> This is a result of BigQueryIO not creating unique job ids for batch inserts, 
> thus BigQuery API responding with a 409 conflict error:
> {code:java}
> Request failed with code 409, will NOT retry: 
> https://www.googleapis.com/bigquery/v2/projects//jobs
> {code}
> The jobs are initiated in a step BatchLoads/SinglePartitionWriteTables, 
> called by step's WriteTables ParDo:
> https://github.com/apache/beam/blob/master/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BatchLoads.java#L511-L521
> https://github.com/apache/beam/blob/master/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/WriteTables.java#L148
> It would probably be a good idea to append a UUIDs as part of a job id.
> Edit: This is a major bug blocking using BigQuery as a sink for bounded input.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (BEAM-2840) BigQueryIO write is slow/fail with a bounded source

[Eugene Kirpichov (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2840?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16340197#comment-16340197
 ] 

Eugene Kirpichov commented on BEAM-2840:


Doing a cleanup pass over BigQuery bugs. Seems this one slipped through the 
cracks. The job's data is not accessible anymore - are you still experiencing 
this with Beam 2.2? If you hit OOMs, try running with --dumpHeapOnOOM=true 
--saveHeapDumpsToGcsPath=gs://my-bucket/dump-directory/ and looking at the heap 
dumps using a memory profiler (my favorite one is Eclipse MAT or YourKit).

> BigQueryIO write is slow/fail with a bounded source
> ---
>
> Key: BEAM-2840
> URL: https://issues.apache.org/jira/browse/BEAM-2840
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-gcp
>Affects Versions: 2.0.0
> Environment: Gougle Cloud Platform
>Reporter: Vincent Spiewak
>Assignee: Reuven Lax
>Priority: Major
> Attachments: PrepareWrite.BatchLoads.png
>
>
> BigQueryIO Writer is slow / fail if the input source is bounded.
> EDIT: Input BQ: 294 GB, 741,896,827 events
> If the input source is bounded (GCS / BQ select / ...), BigQueryIO Writer use 
> the 
> "[Method.FILE_LOADS|https://github.com/apache/beam/blob/master/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryIO.java#L1168];
>  instead of streaming inserts.
> Large amounts of input datas result in a  java.lang.OutOfMemoryError / Java 
> heap space (500 millions rows).
> !PrepareWrite.BatchLoads.png|thumbnail!
> We cannot use "Method.STREAMING_INSERTS" or control the batchs sizes since
> [withMaxFilesPerBundle|https://github.com/apache/beam/blob/master/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryIO.java#L1131]
>  is private :(
> Someone reported a similar problem with GCS -> BQ on Stackoverflow: 
> [Why is writing to BigQuery from a Dataflow/Beam pipeline 
> slow?|https://stackoverflow.com/questions/45889992/why-is-writing-to-bigquery-from-a-dataflow-beam-pipeline-slow#comment78954153_45889992]



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (BEAM-3501) BigQuery Partitioned table creation/write fails when destination has partition decorator

[Chamikara Jayalath (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3501?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chamikara Jayalath reassigned BEAM-3501:


Assignee: Eugene Kirpichov  (was: Chamikara Jayalath)

> BigQuery Partitioned table creation/write fails when destination has 
> partition decorator
> 
>
> Key: BEAM-3501
> URL: https://issues.apache.org/jira/browse/BEAM-3501
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-gcp
>Reporter: Darshan Mehta
>Assignee: Eugene Kirpichov
>Priority: Major
>
> Following is the code that writes to BigQuery: 
> {code:java}
> BigQueryIO.writeTableRows()
>  .to(destination)
>  .withCreateDisposition(CREATE_IF_NEEDED)
>  .withWriteDisposition(WRITE_APPEND)
>  .withSchema(tableSchema)
>  .expand(tableRows);{code}
>  
> Here's the destination's implementation: 
> {code:java}
> public TableDestination apply(ValueInSingleWindow input) {
>  String partition = timestampExtractor.apply(input.getValue())
>  .toString(DateTimeFormat.forPattern("MMdd").withZoneUTC());
>  TableReference tableReference = new TableReference();
>  tableReference.setDatasetId(dataset);
>  tableReference.setProjectId(projectId);
>  tableReference.setTableId(String.format("%s_%s", table, partition));
>  log.debug("Will write to BigQuery table: %s", tableReference);
>  return new TableDestination(tableReference, null);
> }{code}
>  
> When the dataflow tries to write to this table, I see the following message:
> {code:java}
> "errors" : [ {
>  "domain" : "global",
>  "message" : "Cannot read partition information from a table that is not 
> partitioned: :.$19730522",
>  "reason" : "invalid"
>  } ]{code}
> So, it looks like it's not creating tables with partition in the first place? 
> Apache beam version : 2.2.0



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[beam] 01/01: Merge pull request #4417 from shashank64/master

[jkff]

This is an automated email from the ASF dual-hosted git repository.

jkff pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git

commit 0f554fd51414381646c32b36f6f87a14adc0fad8
Merge: 5ee938e aaabb64
Author: Eugene Kirpichov 
AuthorDate: Thu Jan 25 14:17:27 2018 -0800

Merge pull request #4417 from shashank64/master

[BEAM-3482] fix serialization error in BigQueryIO's DynamicDestinations

 .../sdk/io/gcp/bigquery/DynamicDestinations.java   | 24 --
 1 file changed, 13 insertions(+), 11 deletions(-)

-- 
To stop receiving notification emails like this one, please contact
j...@apache.org.

[beam] branch master updated (5ee938e -> 0f554fd)

[jkff]

This is an automated email from the ASF dual-hosted git repository.

jkff pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from 5ee938e  Merge pull request #4489: [BEAM-3466] Remove JDK 7 references 
in Jenkins
 add 7061301  fix serialization error in BigQueryIO's DynamicDestinations
 add aaabb64  correct side input check in BigQueryIO DynamicDestination for 
pull
 new 0f554fd  Merge pull request #4417 from shashank64/master

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 .../sdk/io/gcp/bigquery/DynamicDestinations.java   | 24 --
 1 file changed, 13 insertions(+), 11 deletions(-)

-- 
To stop receiving notification emails like this one, please contact
j...@apache.org.

Jenkins build is back to normal : beam_PostCommit_Python_Verify #4056

[Apache Jenkins Server]

See 

[jira] [Commented] (BEAM-3501) BigQuery Partitioned table creation/write fails when destination has partition decorator

[Eugene Kirpichov (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-3501?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16340173#comment-16340173
 ] 

Eugene Kirpichov commented on BEAM-3501:


I tried doing a similar pipeline myself:
 * Writing to table with partition decorators using DynamicDestinations, where 
the table does not exist. It succeeds if I specify the TimePartitioning in the 
TableDestination's in getTable(); and if I don't, it fails with a different 
error (asking to specify TimePartitioning).
 * Same but the table exists and is unpartitioned. Then I get yet a different 
error: "Cannot add storage to a non-partitioned table with a partition 
reference:..."

So far I'm unable to reproduce your issue. I'd really appreciate more details 
here.

 

One issue I see is that you're using .expand() explicitly - this is incorrect 
and can lead to all sorts of issues. expand() is an implementation detail of 
all transforms, users MUST use .apply() instead.

> BigQuery Partitioned table creation/write fails when destination has 
> partition decorator
> 
>
> Key: BEAM-3501
> URL: https://issues.apache.org/jira/browse/BEAM-3501
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-gcp
>Reporter: Darshan Mehta
>Assignee: Chamikara Jayalath
>Priority: Major
>
> Following is the code that writes to BigQuery: 
> {code:java}
> BigQueryIO.writeTableRows()
>  .to(destination)
>  .withCreateDisposition(CREATE_IF_NEEDED)
>  .withWriteDisposition(WRITE_APPEND)
>  .withSchema(tableSchema)
>  .expand(tableRows);{code}
>  
> Here's the destination's implementation: 
> {code:java}
> public TableDestination apply(ValueInSingleWindow input) {
>  String partition = timestampExtractor.apply(input.getValue())
>  .toString(DateTimeFormat.forPattern("MMdd").withZoneUTC());
>  TableReference tableReference = new TableReference();
>  tableReference.setDatasetId(dataset);
>  tableReference.setProjectId(projectId);
>  tableReference.setTableId(String.format("%s_%s", table, partition));
>  log.debug("Will write to BigQuery table: %s", tableReference);
>  return new TableDestination(tableReference, null);
> }{code}
>  
> When the dataflow tries to write to this table, I see the following message:
> {code:java}
> "errors" : [ {
>  "domain" : "global",
>  "message" : "Cannot read partition information from a table that is not 
> partitioned: :.$19730522",
>  "reason" : "invalid"
>  } ]{code}
> So, it looks like it's not creating tables with partition in the first place? 
> Apache beam version : 2.2.0



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Jenkins build is still unstable: beam_PostCommit_Java_MavenInstall #5759

[Apache Jenkins Server]

See 

[jira] [Closed] (BEAM-3311) Extend BigTableIO to write Iterable of KV

[Solomon Duskis (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3311?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Solomon Duskis closed BEAM-3311.

   Resolution: Won't Fix
Fix Version/s: Not applicable

Use Flatten.iterable() instead of duplicating that functionality in BigtableIO.

> Extend BigTableIO to write Iterable of KV 
> --
>
> Key: BEAM-3311
> URL: https://issues.apache.org/jira/browse/BEAM-3311
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-gcp
>Affects Versions: 2.2.0
>Reporter: Anna Smith
>Assignee: Solomon Duskis
>Priority: Major
> Fix For: Not applicable
>
>
> The motivation is to achieve qps as advertised in BigTable in Dataflow 
> streaming mode (ex: 300k qps for 30 node cluster).  Currently we aren't 
> seeing this as the bundle size is small in streaming mode and the requests 
> are overwhelmed by AuthentiationHeader.  For example, in order to achieve qps 
> advertised each payload is recommended to be ~1KB but without batching each 
> payload is 7KB, the majority of which is the authentication header.
> Currently BigTableIO supports DoFn,...> 
> where batching is done per Bundle on flush in finishBundle. We would like to 
> be able to manually batch using a DoFn Iterable>>,...> so we can get around the small Bundle size in 
> streaming.  We have seen some improvements in qps to BigTable when running 
> with Dataflow using this approach.
> Initial thoughts on implementation would be to extend Write in order to have 
> a BulkWrite of Iterable>.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (BEAM-3311) Extend BigTableIO to write Iterable of KV

[Solomon Duskis (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-3311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16340124#comment-16340124
 ] 

Solomon Duskis commented on BEAM-3311:
--

I spoke quite a bit with the Beam team about this.  BigtableIO should remain as 
is.  It looks like there's a _Flatten.iterables()_ which ought to convert an 
_Iterable_ to a _T_.  The BigtableIO connector is meant to satisfy 80%+ of 
the use cases.  In other cases, I generally look for common usage patterns 
before a change is made to any connector.  

In addition to this approach, you can also create your own DoFn that does 
arbitrary operations against a 
[BigtableSession|https://github.com/GoogleCloudPlatform/cloud-bigtable-client/blob/master/bigtable-client-core-parent/bigtable-client-core/src/main/java/com/google/cloud/bigtable/grpc/BigtableSession.java].
  Be sure to use _BigtableOptions.Builder.setUseCachedDataPool(true)_, if you 
chose to go down this route.

> Extend BigTableIO to write Iterable of KV 
> --
>
> Key: BEAM-3311
> URL: https://issues.apache.org/jira/browse/BEAM-3311
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-gcp
>Affects Versions: 2.2.0
>Reporter: Anna Smith
>Assignee: Solomon Duskis
>Priority: Major
>
> The motivation is to achieve qps as advertised in BigTable in Dataflow 
> streaming mode (ex: 300k qps for 30 node cluster).  Currently we aren't 
> seeing this as the bundle size is small in streaming mode and the requests 
> are overwhelmed by AuthentiationHeader.  For example, in order to achieve qps 
> advertised each payload is recommended to be ~1KB but without batching each 
> payload is 7KB, the majority of which is the authentication header.
> Currently BigTableIO supports DoFn,...> 
> where batching is done per Bundle on flush in finishBundle. We would like to 
> be able to manually batch using a DoFn Iterable>>,...> so we can get around the small Bundle size in 
> streaming.  We have seen some improvements in qps to BigTable when running 
> with Dataflow using this approach.
> Initial thoughts on implementation would be to extend Write in order to have 
> a BulkWrite of Iterable>.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Jenkins build is back to normal : beam_PostCommit_Python_ValidatesRunner_Dataflow #736

[Apache Jenkins Server]

See 

Build failed in Jenkins: beam_PostCommit_Python_ValidatesContainer_Dataflow #11

[Apache Jenkins Server]

See 


Changes:

[jbonofre] [BEAM-3466] Remove JDK 7 references in Jenkins

--
Started by GitHub push by kennknowles
[EnvInject] - Loading node environment variables.
Building remotely on beam1 (beam) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* 
 > +refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/*
 > git rev-parse origin/master^{commit} # timeout=10
Checking out Revision 5ee938ec37f729f5fca96e2b02b034505fc86fb0 (origin/master)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 5ee938ec37f729f5fca96e2b02b034505fc86fb0
Commit message: "Merge pull request #4489: [BEAM-3466] Remove JDK 7 references 
in Jenkins"
 > git rev-list e28c4da27f781fa38edebfe3140a71b921aaa242 # timeout=10
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
[EnvInject] - Executing scripts and injecting environment variables after the 
SCM step.
[EnvInject] - Injecting as environment variables the properties content 
SPARK_LOCAL_IP=127.0.0.1

[EnvInject] - Variables injected successfully.
[beam_PostCommit_Python_ValidatesContainer_Dataflow] $ /bin/bash -xe 
/tmp/jenkins7927482264280354739.sh
+ cd src
+ bash sdks/python/container/run_validatescontainer.sh
bash: sdks/python/container/run_validatescontainer.sh: No such file or directory
Build step 'Execute shell' marked build as failure
Not sending mail to unregistered user rob...@frantil.com

[jira] [Commented] (BEAM-3484) HadoopInputFormatIO reads big datasets invalid

[JIRA]

[ 
https://issues.apache.org/jira/browse/BEAM-3484?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16340099#comment-16340099
 ] 

Ismaël Mejía commented on BEAM-3484:


I am reasigning this to [~aromanenko] who is going to help me take a look on it.

> HadoopInputFormatIO reads big datasets invalid
> --
>
> Key: BEAM-3484
> URL: https://issues.apache.org/jira/browse/BEAM-3484
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-extensions
>Reporter: Łukasz Gajowy
>Assignee: Alexey Romanenko
>Priority: Major
> Attachments: result_sorted100, result_sorted60
>
>
> For big datasets HadoopInputFormat sometimes skips/duplicates elements from 
> database in resulting PCollection. This gives incorrect read result.
> Occurred to me while developing HadoopInputFormatIOIT and running it on 
> dataflow. For datasets smaller or equal to 600 000 database rows I wasn't 
> able to reproduce the issue. Bug appeared only for bigger sets, eg. 700 000, 
> 1 000 000. 
> Attachments:
>   - text file with sorted HadoopInputFormat.read() result saved using 
> TextIO.write().to().withoutSharding(). If you look carefully you'll notice 
> duplicates or missing values that should not happen
>  - same text file for 600 000 records not having any duplicates and missing 
> elements
>  - link to a PR with HadoopInputFormatIO integration test that allows to 
> reproduce this issue. At the moment of writing, this code is not merged yet.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (BEAM-3484) HadoopInputFormatIO reads big datasets invalid

[JIRA]

 [ 
https://issues.apache.org/jira/browse/BEAM-3484?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismaël Mejía reassigned BEAM-3484:
--

Assignee: Alexey Romanenko  (was: Ismaël Mejía)

> HadoopInputFormatIO reads big datasets invalid
> --
>
> Key: BEAM-3484
> URL: https://issues.apache.org/jira/browse/BEAM-3484
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-extensions
>Reporter: Łukasz Gajowy
>Assignee: Alexey Romanenko
>Priority: Major
> Attachments: result_sorted100, result_sorted60
>
>
> For big datasets HadoopInputFormat sometimes skips/duplicates elements from 
> database in resulting PCollection. This gives incorrect read result.
> Occurred to me while developing HadoopInputFormatIOIT and running it on 
> dataflow. For datasets smaller or equal to 600 000 database rows I wasn't 
> able to reproduce the issue. Bug appeared only for bigger sets, eg. 700 000, 
> 1 000 000. 
> Attachments:
>   - text file with sorted HadoopInputFormat.read() result saved using 
> TextIO.write().to().withoutSharding(). If you look carefully you'll notice 
> duplicates or missing values that should not happen
>  - same text file for 600 000 records not having any duplicates and missing 
> elements
>  - link to a PR with HadoopInputFormatIO integration test that allows to 
> reproduce this issue. At the moment of writing, this code is not merged yet.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (BEAM-3533) Replace hard-coded UTF-8 Strings

[JIRA]

 [ 
https://issues.apache.org/jira/browse/BEAM-3533?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismaël Mejía reassigned BEAM-3533:
--

Assignee: Colm O hEigeartaigh  (was: Jean-Baptiste Onofré)

> Replace hard-coded UTF-8 Strings
> 
>
> Key: BEAM-3533
> URL: https://issues.apache.org/jira/browse/BEAM-3533
> Project: Beam
>  Issue Type: Improvement
>  Components: beam-model
>Reporter: Colm O hEigeartaigh
>Assignee: Colm O hEigeartaigh
>Priority: Trivial
> Fix For: 2.3.0
>
>
> This task is to replace hard-coded UTF-8 Strings in the code with 
> StandardCharsets.UTF_8 instead.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Jenkins build is still unstable: beam_PostCommit_Java_ValidatesRunner_Spark #3993

[Apache Jenkins Server]

See 

[jira] [Resolved] (BEAM-3466) Remove Java 7 and any related task from Jenkins

[JIRA]

 [ 
https://issues.apache.org/jira/browse/BEAM-3466?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jean-Baptiste Onofré resolved BEAM-3466.

Resolution: Fixed

> Remove Java 7 and any related task from Jenkins
> ---
>
> Key: BEAM-3466
> URL: https://issues.apache.org/jira/browse/BEAM-3466
> Project: Beam
>  Issue Type: Sub-task
>  Components: build-system
>Reporter: Ismaël Mejía
>Assignee: Jean-Baptiste Onofré
>Priority: Major
> Fix For: 2.3.0
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Jenkins build became unstable: beam_PostCommit_Java_MavenInstall #5758

[Apache Jenkins Server]

See 

[jira] [Commented] (BEAM-3466) Remove Java 7 and any related task from Jenkins

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-3466?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16339847#comment-16339847
 ] 

ASF GitHub Bot commented on BEAM-3466:
--

kennknowles closed pull request #4489: [BEAM-3466] Remove JDK 7 references in 
Jenkins
URL: https://github.com/apache/beam/pull/4489
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git 
a/.test-infra/jenkins/job_beam_PostCommit_Java_JDKVersionsTest.groovy 
b/.test-infra/jenkins/job_beam_PostCommit_Java_JDKVersionsTest.groovy
index df0a2c7a6d3..a2f6aaf5052 100644
--- a/.test-infra/jenkins/job_beam_PostCommit_Java_JDKVersionsTest.groovy
+++ b/.test-infra/jenkins/job_beam_PostCommit_Java_JDKVersionsTest.groovy
@@ -28,9 +28,7 @@ matrixJob('beam_PostCommit_Java_JDK_Versions_Test') {
   // Set JDK versions.
   axes {
 label('label', 'beam')
-jdk('JDK 1.7 (latest)',
-'OpenJDK 7 (on Ubuntu only)',
-'OpenJDK 8 (on Ubuntu only)')
+jdk('OpenJDK 8 (on Ubuntu only)')
   }
 
   // Sets that this is a PostCommit job.


 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Remove Java 7 and any related task from Jenkins
> ---
>
> Key: BEAM-3466
> URL: https://issues.apache.org/jira/browse/BEAM-3466
> Project: Beam
>  Issue Type: Sub-task
>  Components: build-system
>Reporter: Ismaël Mejía
>Assignee: Jean-Baptiste Onofré
>Priority: Major
> Fix For: 2.3.0
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[beam] 01/01: Merge pull request #4489: [BEAM-3466] Remove JDK 7 references in Jenkins

[kenn]

This is an automated email from the ASF dual-hosted git repository.

kenn pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git

commit 5ee938ec37f729f5fca96e2b02b034505fc86fb0
Merge: e28c4da dafe47d
Author: Kenn Knowles 
AuthorDate: Thu Jan 25 12:42:09 2018 -0800

Merge pull request #4489: [BEAM-3466] Remove JDK 7 references in Jenkins

 .test-infra/jenkins/job_beam_PostCommit_Java_JDKVersionsTest.groovy | 4 +---
 1 file changed, 1 insertion(+), 3 deletions(-)

-- 
To stop receiving notification emails like this one, please contact
k...@apache.org.

[beam] branch master updated (e28c4da -> 5ee938e)

[kenn]

This is an automated email from the ASF dual-hosted git repository.

kenn pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from e28c4da  Merge pull request #4486 from coheigea/BEAM-3533
 add dafe47d  [BEAM-3466] Remove JDK 7 references in Jenkins
 new 5ee938e  Merge pull request #4489: [BEAM-3466] Remove JDK 7 references 
in Jenkins

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 .test-infra/jenkins/job_beam_PostCommit_Java_JDKVersionsTest.groovy | 4 +---
 1 file changed, 1 insertion(+), 3 deletions(-)

-- 
To stop receiving notification emails like this one, please contact
k...@apache.org.

[jira] [Assigned] (BEAM-3519) GCP IO exposes netty on its API surface, causing conflicts with runners

[JIRA]

 [ 
https://issues.apache.org/jira/browse/BEAM-3519?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismaël Mejía reassigned BEAM-3519:
--

Assignee: (was: Reuven Lax)

> GCP IO exposes netty on its API surface, causing conflicts with runners
> ---
>
> Key: BEAM-3519
> URL: https://issues.apache.org/jira/browse/BEAM-3519
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-gcp
>Reporter: Ismaël Mejía
>Priority: Critical
>
> Google Cloud Platform IOs module leaks netty this causes conflicts in 
> particular with execution systems that use conflicting versions of such 
> modules. 
>  For the case there is a dependency conflict with the Spark Runner version of 
> netty, see: BEAM-3492



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (BEAM-2523) GCP IO exposes protobuf on its API surface, causing user pain

[JIRA]

 [ 
https://issues.apache.org/jira/browse/BEAM-2523?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismaël Mejía updated BEAM-2523:
---
Component/s: (was: sdk-java-extensions)
 sdk-java-gcp

> GCP IO exposes protobuf on its API surface, causing user pain
> -
>
> Key: BEAM-2523
> URL: https://issues.apache.org/jira/browse/BEAM-2523
> Project: Beam
>  Issue Type: Bug
>  Components: runner-dataflow, sdk-java-gcp
>Reporter: Kenneth Knowles
>Priority: Critical
>
> Putting the SDK, DataflowRunner, and GCP IO on the same classpath, results in 
> (at least) three versions of protobuf getting pulled in. These should be made 
> to converge. We should consider using maven enforcer, which I think can check 
> this.
> {code}
> [INFO] com.example:foo:jar:0.1
> [INFO] +- org.apache.beam:beam-sdks-java-core:jar:2.0.0:compile
> [INFO] +- 
> org.apache.beam:beam-sdks-java-io-google-cloud-platform:jar:2.0.0:compile
> [INFO] |  +- 
> org.apache.beam:beam-sdks-java-extensions-protobuf:jar:2.0.0:compile
> [INFO] |  |  \- (com.google.protobuf:protobuf-java:jar:3.2.0:compile - 
> omitted for duplicate)
> [INFO] |  +- com.google.api.grpc:grpc-google-pubsub-v1:jar:0.1.0:compile
> [INFO] |  |  +- (com.google.protobuf:protobuf-java:jar:3.0.0:compile - 
> omitted for conflict with 3.2.0)
> [INFO] |  |  \- com.google.api.grpc:grpc-google-iam-v1:jar:0.1.0:compile
> [INFO] |  | \- (com.google.protobuf:protobuf-java:jar:3.0.0:compile - 
> omitted for conflict with 3.2.0)
> [INFO] |  +- 
> com.google.cloud.datastore:datastore-v1-proto-client:jar:1.4.0:compile
> [INFO] |  |  +- 
> (com.google.cloud.datastore:datastore-v1-protos:jar:1.3.0:compile - omitted 
> for duplicate)
> [INFO] |  |  +- (com.google.http-client:google-http-client:jar:1.20.0:compile 
> - omitted for conflict with 1.22.0)
> [INFO] |  |  +- 
> com.google.http-client:google-http-client-protobuf:jar:1.20.0:compile
> [INFO] |  |  |  +- 
> (com.google.http-client:google-http-client:jar:1.20.0:compile - omitted for 
> conflict with 1.22.0)
> [INFO] |  |  |  \- (com.google.protobuf:protobuf-java:jar:2.4.1:compile - 
> omitted for conflict with 3.2.0)
> [INFO] |  +- com.google.cloud.datastore:datastore-v1-protos:jar:1.3.0:compile
> [INFO] |  |  +- (com.google.protobuf:protobuf-java:jar:3.0.0:compile - 
> omitted for conflict with 3.2.0)
> [INFO] |  +- com.google.cloud.bigtable:bigtable-protos:jar:0.9.6.2:compile
> [INFO] |  |  +- (com.google.code.findbugs:jsr305:jar:3.0.1:compile - omitted 
> for duplicate)
> [INFO] |  |  +- (com.google.protobuf:protobuf-java:jar:3.2.0:compile - 
> omitted for duplicate)
> {code}
> Incidentally, the dependency plugin stopped supporting the verbose tree, so 
> we can't even visually inspect this except by downgrading.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (BEAM-3519) GCP IO exposes netty on its API surface, causing conflicts with runners

[JIRA]

 [ 
https://issues.apache.org/jira/browse/BEAM-3519?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismaël Mejía updated BEAM-3519:
---
Component/s: (was: sdk-java-extensions)
 sdk-java-gcp

> GCP IO exposes netty on its API surface, causing conflicts with runners
> ---
>
> Key: BEAM-3519
> URL: https://issues.apache.org/jira/browse/BEAM-3519
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-gcp
>Reporter: Ismaël Mejía
>Assignee: Reuven Lax
>Priority: Critical
>
> Google Cloud Platform IOs module leaks netty this causes conflicts in 
> particular with execution systems that use conflicting versions of such 
> modules. 
>  For the case there is a dependency conflict with the Spark Runner version of 
> netty, see: BEAM-3492



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (BEAM-812) GCP IO exposes guava on its API surface, causing user pain

[JIRA]

 [ 
https://issues.apache.org/jira/browse/BEAM-812?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismaël Mejía updated BEAM-812:
--
Summary: GCP IO exposes guava on its API surface, causing user pain  (was: 
Shade guava in beam-sdks-java-io-google-cloud-platform)

> GCP IO exposes guava on its API surface, causing user pain
> --
>
> Key: BEAM-812
> URL: https://issues.apache.org/jira/browse/BEAM-812
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-gcp
>Reporter: Daniel Halperin
>Priority: Major
>
> Looking at 0.3.0-incubating RC1, we are not properly shading Guava.
> https://repository.apache.org/content/repositories/staging/org/apache/beam/beam-sdks-java-io-google-cloud-platform/0.3.0-incubating/beam-sdks-java-io-google-cloud-platform-0.3.0-incubating.pom
> has 
> {code}
> 
>   com.google.guava
>   guava
>  {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (BEAM-812) GCP IO exposes guava on its API surface, causing user pain

[JIRA]

 [ 
https://issues.apache.org/jira/browse/BEAM-812?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismaël Mejía updated BEAM-812:
--
Priority: Critical  (was: Major)

> GCP IO exposes guava on its API surface, causing user pain
> --
>
> Key: BEAM-812
> URL: https://issues.apache.org/jira/browse/BEAM-812
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-gcp
>Reporter: Daniel Halperin
>Priority: Critical
>
> Looking at 0.3.0-incubating RC1, we are not properly shading Guava.
> https://repository.apache.org/content/repositories/staging/org/apache/beam/beam-sdks-java-io-google-cloud-platform/0.3.0-incubating/beam-sdks-java-io-google-cloud-platform-0.3.0-incubating.pom
> has 
> {code}
> 
>   com.google.guava
>   guava
>  {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (BEAM-3519) GCP IO exposes netty on its API surface, causing conflicts with runners

[JIRA]

 [ 
https://issues.apache.org/jira/browse/BEAM-3519?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismaël Mejía updated BEAM-3519:
---
Description: 
Google Cloud Platform IOs module leaks netty this causes conflicts in 
particular with execution systems that use conflicting versions of such 
modules. 
 For the case there is a dependency conflict with the Spark Runner version of 
netty, see: BEAM-3492

  was:
Google Cloud Platform IOs module leaks some of the classical core dependencies 
(netty/protobuf) and this can cause conflicts in particular with execution 
systems that use conflicting versions of such modules. 
For the case there is a dependency conflict with the Spark Runner version of 
netty, see: BEAM-3492


> GCP IO exposes netty on its API surface, causing conflicts with runners
> ---
>
> Key: BEAM-3519
> URL: https://issues.apache.org/jira/browse/BEAM-3519
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-extensions
>Reporter: Ismaël Mejía
>Assignee: Reuven Lax
>Priority: Critical
>
> Google Cloud Platform IOs module leaks netty this causes conflicts in 
> particular with execution systems that use conflicting versions of such 
> modules. 
>  For the case there is a dependency conflict with the Spark Runner version of 
> netty, see: BEAM-3492



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (BEAM-2523) GCP IO exposes protobuf on its API surface, causing user pain

[JIRA]

 [ 
https://issues.apache.org/jira/browse/BEAM-2523?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismaël Mejía updated BEAM-2523:
---
Component/s: sdk-java-extensions

> GCP IO exposes protobuf on its API surface, causing user pain
> -
>
> Key: BEAM-2523
> URL: https://issues.apache.org/jira/browse/BEAM-2523
> Project: Beam
>  Issue Type: Bug
>  Components: runner-dataflow, sdk-java-extensions
>Reporter: Kenneth Knowles
>Priority: Critical
>
> Putting the SDK, DataflowRunner, and GCP IO on the same classpath, results in 
> (at least) three versions of protobuf getting pulled in. These should be made 
> to converge. We should consider using maven enforcer, which I think can check 
> this.
> {code}
> [INFO] com.example:foo:jar:0.1
> [INFO] +- org.apache.beam:beam-sdks-java-core:jar:2.0.0:compile
> [INFO] +- 
> org.apache.beam:beam-sdks-java-io-google-cloud-platform:jar:2.0.0:compile
> [INFO] |  +- 
> org.apache.beam:beam-sdks-java-extensions-protobuf:jar:2.0.0:compile
> [INFO] |  |  \- (com.google.protobuf:protobuf-java:jar:3.2.0:compile - 
> omitted for duplicate)
> [INFO] |  +- com.google.api.grpc:grpc-google-pubsub-v1:jar:0.1.0:compile
> [INFO] |  |  +- (com.google.protobuf:protobuf-java:jar:3.0.0:compile - 
> omitted for conflict with 3.2.0)
> [INFO] |  |  \- com.google.api.grpc:grpc-google-iam-v1:jar:0.1.0:compile
> [INFO] |  | \- (com.google.protobuf:protobuf-java:jar:3.0.0:compile - 
> omitted for conflict with 3.2.0)
> [INFO] |  +- 
> com.google.cloud.datastore:datastore-v1-proto-client:jar:1.4.0:compile
> [INFO] |  |  +- 
> (com.google.cloud.datastore:datastore-v1-protos:jar:1.3.0:compile - omitted 
> for duplicate)
> [INFO] |  |  +- (com.google.http-client:google-http-client:jar:1.20.0:compile 
> - omitted for conflict with 1.22.0)
> [INFO] |  |  +- 
> com.google.http-client:google-http-client-protobuf:jar:1.20.0:compile
> [INFO] |  |  |  +- 
> (com.google.http-client:google-http-client:jar:1.20.0:compile - omitted for 
> conflict with 1.22.0)
> [INFO] |  |  |  \- (com.google.protobuf:protobuf-java:jar:2.4.1:compile - 
> omitted for conflict with 3.2.0)
> [INFO] |  +- com.google.cloud.datastore:datastore-v1-protos:jar:1.3.0:compile
> [INFO] |  |  +- (com.google.protobuf:protobuf-java:jar:3.0.0:compile - 
> omitted for conflict with 3.2.0)
> [INFO] |  +- com.google.cloud.bigtable:bigtable-protos:jar:0.9.6.2:compile
> [INFO] |  |  +- (com.google.code.findbugs:jsr305:jar:3.0.1:compile - omitted 
> for duplicate)
> [INFO] |  |  +- (com.google.protobuf:protobuf-java:jar:3.2.0:compile - 
> omitted for duplicate)
> {code}
> Incidentally, the dependency plugin stopped supporting the verbose tree, so 
> we can't even visually inspect this except by downgrading.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (BEAM-3519) GCP IO exposes netty on its API surface, causing conflicts with runners

[JIRA]

 [ 
https://issues.apache.org/jira/browse/BEAM-3519?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismaël Mejía updated BEAM-3519:
---
Summary: GCP IO exposes netty on its API surface, causing conflicts with 
runners  (was: Shade netty and protobuf in 
beam-sdks-java-io-google-cloud-platform)

> GCP IO exposes netty on its API surface, causing conflicts with runners
> ---
>
> Key: BEAM-3519
> URL: https://issues.apache.org/jira/browse/BEAM-3519
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-extensions
>Reporter: Ismaël Mejía
>Assignee: Reuven Lax
>Priority: Critical
>
> Google Cloud Platform IOs module leaks some of the classical core 
> dependencies (netty/protobuf) and this can cause conflicts in particular with 
> execution systems that use conflicting versions of such modules. 
> For the case there is a dependency conflict with the Spark Runner version of 
> netty, see: BEAM-3492



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (BEAM-3519) Shade netty and protobuf in beam-sdks-java-io-google-cloud-platform

[JIRA]

 [ 
https://issues.apache.org/jira/browse/BEAM-3519?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismaël Mejía updated BEAM-3519:
---
Issue Type: Bug  (was: Improvement)

> Shade netty and protobuf in beam-sdks-java-io-google-cloud-platform
> ---
>
> Key: BEAM-3519
> URL: https://issues.apache.org/jira/browse/BEAM-3519
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-extensions
>Reporter: Ismaël Mejía
>Assignee: Reuven Lax
>Priority: Major
>
> Google Cloud Platform IOs module leaks some of the classical core 
> dependencies (netty/protobuf) and this can cause conflicts in particular with 
> execution systems that use conflicting versions of such modules. 
> For the case there is a dependency conflict with the Spark Runner version of 
> netty, see: BEAM-3492



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (BEAM-3519) Shade netty and protobuf in beam-sdks-java-io-google-cloud-platform

[JIRA]

 [ 
https://issues.apache.org/jira/browse/BEAM-3519?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismaël Mejía updated BEAM-3519:
---
Priority: Critical  (was: Major)

> Shade netty and protobuf in beam-sdks-java-io-google-cloud-platform
> ---
>
> Key: BEAM-3519
> URL: https://issues.apache.org/jira/browse/BEAM-3519
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-extensions
>Reporter: Ismaël Mejía
>Assignee: Reuven Lax
>Priority: Critical
>
> Google Cloud Platform IOs module leaks some of the classical core 
> dependencies (netty/protobuf) and this can cause conflicts in particular with 
> execution systems that use conflicting versions of such modules. 
> For the case there is a dependency conflict with the Spark Runner version of 
> netty, see: BEAM-3492



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (BEAM-3152) AfterProcessingTime trigger doesn't create any file panes

[Eugene Kirpichov (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-3152?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16339835#comment-16339835
 ] 

Eugene Kirpichov commented on BEAM-3152:


Does this issue affect Beam 2.2 or at HEAD? The WriteFiles transform had a 
windowing bug that has since been fixed.

> AfterProcessingTime trigger doesn't create any file panes
> -
>
> Key: BEAM-3152
> URL: https://issues.apache.org/jira/browse/BEAM-3152
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Affects Versions: 2.0.0
>Reporter: Pawel Bartoszek
>Assignee: Eugene Kirpichov
>Priority: Major
>
> Scenario:
> I want to count how many events A and B I am getting for given 30 window. I 
> require that every pane has all events types (A and B) with corresponding 
> counters - this is why I am using Combine.globally.
> The calculation logic works fine the problem is with writing files. The files 
> are not written.
> For debugging purposes I created some transformations (Simulate 
> ApplyShardLabel, Simulate GroupIntoShards etc) that mimics that logic 
> implemented by WriteFiles.
> If you push string "A" and "B" to kinesis stream I am seeing the following 
> system.out from the job:
> {code:java}
> AFTER COMBINE: {A=1, B=1}
> {code}
> According to my test transformations I should also see:
> {code:java}
> AFTER COMBINE: {A=1, B=1}
> Simulating ApplyShardLabel
> Simulating finalizing writer: KV{null, [KV{0, [{A=1, B=1}]}]}
> {code}
> Using DirectRunner and Beam 2.0.0. When I switch to Beam 2.1.0 I see the 
> expected debug output and files being written out.
> I think that there is some issue with AfterSynchronizedProcessingTime trigger 
> support.
> I cannot replicate the issue when using `TestStream`
> The test code can be found at
> [https://gist.github.com/pbartoszek/9dd58c4fcfc5171eafba3520cb3040fa|https://gist.github.com/pbartoszek/9dd58c4fcfc5171eafba3520cb3040fa]



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Closed] (BEAM-730) Remove Reshuffle transform in favor of Redistribute.byKey()

[Eugene Kirpichov (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-730?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Eugene Kirpichov closed BEAM-730.
-
   Resolution: Won't Fix
Fix Version/s: Not applicable

We have Reshuffle.viaRandomKey() for one case, and RequiresStableInput 
annotation for another (not implemented fully yet) and no substantial progress 
being made other than that. Discussions about Redistribute ended in concluding 
that there's no single transform to be implemented here.

> Remove Reshuffle transform in favor of Redistribute.byKey()
> ---
>
> Key: BEAM-730
> URL: https://issues.apache.org/jira/browse/BEAM-730
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Reporter: Eugene Kirpichov
>Assignee: Eugene Kirpichov
>Priority: Major
> Fix For: Not applicable
>
>
> https://github.com/apache/incubator-beam/pull/1036 introduces a Redistribute 
> transform which is a superset of Reshuffle. Reshuffle is an internal 
> implementation detail transform, while Redistribute is generally useful. The 
> current bug is to remove Reshuffle or hide it from users to reduce confusion.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Closed] (BEAM-2734) Dataflow ValidatesRunner broken at HEAD

[Eugene Kirpichov (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2734?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Eugene Kirpichov closed BEAM-2734.
--
   Resolution: Cannot Reproduce
Fix Version/s: Not applicable

This was fixed a while ago.

> Dataflow ValidatesRunner broken at HEAD
> ---
>
> Key: BEAM-2734
> URL: https://issues.apache.org/jira/browse/BEAM-2734
> Project: Beam
>  Issue Type: Bug
>  Components: runner-dataflow
>Reporter: Eugene Kirpichov
>Assignee: Eugene Kirpichov
>Priority: Critical
> Fix For: Not applicable
>
>
> https://builds.apache.org/view/Beam/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/
> Last stable build (#3567), 23 days ago
> We need to fix this.
> One of the issues is something I broke recently myself: SplittableDoFnTest is 
> broken due to serialization incompatibility by 
> https://github.com/apache/beam/pull/3649 which slightly changed that 
> BoundedToUnboundedSourceAdapter class, which changed its serialVersionUID. I 
> don't know why this is so fragile - we should also fix the fact that it is so 
> fragile.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Closed] (BEAM-3267) Return file names from TFRecordIO write

[Eugene Kirpichov (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3267?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Eugene Kirpichov closed BEAM-3267.
--
   Resolution: Fixed
Fix Version/s: 2.3.0

FileIO.write() is in, and support for it in TFRecordIO is in as well.

> Return file names from TFRecordIO write
> ---
>
> Key: BEAM-3267
> URL: https://issues.apache.org/jira/browse/BEAM-3267
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-core
>Reporter: Łukasz Gajowy
>Assignee: Eugene Kirpichov
>Priority: Minor
> Fix For: 2.3.0
>
>
> In TFRecordIO we cannot actually retrurn filename set while doing the write. 
> It prevents us from creating pipelines that write and then read in the same 
> pipeline. It was acomplished in TextIO - there's even an example use case in 
> TextIOIT. Maybe we can do the same in TFRecordIO?
> See: 
> https://github.com/apache/beam/blob/f8d8ff14c49e4dfb15541f4b73aa66513c9a9d23/sdks/java/io/file-based-io-tests/src/test/java/org/apache/beam/sdk/io/text/TextIOIT.java#L114



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Closed] (BEAM-2844) Support implicit side inputs

[Eugene Kirpichov (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2844?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Eugene Kirpichov closed BEAM-2844.
--
   Resolution: Won't Fix
Fix Version/s: 2.2.0

This was superseded by [http://s.apache.org/context-fn] .

> Support implicit side inputs
> 
>
> Key: BEAM-2844
> URL: https://issues.apache.org/jira/browse/BEAM-2844
> Project: Beam
>  Issue Type: Bug
>  Components: runner-core, sdk-java-core
>Reporter: Eugene Kirpichov
>Assignee: Eugene Kirpichov
>Priority: Major
> Fix For: 2.2.0
>
>
> https://lists.apache.org/thread.html/cc543556cc709a44ed92262207215eaa0e43a0f573c630b6360d4edc@%3Cdev.beam.apache.org%3E
> Implement access to side inputs from arbitrary code via PCollectionView.get().
> Not necessarily in scope of this bug: implement inference of which side 
> inputs are needed by detecting fields with a @SideInput annotation.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (BEAM-3430) Update website documentation about Apache Beam 2.3.0+ being Java 8

[JIRA]

 [ 
https://issues.apache.org/jira/browse/BEAM-3430?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jean-Baptiste Onofré updated BEAM-3430:
---
Fix Version/s: 2.3.0

> Update website documentation about Apache Beam 2.3.0+ being Java 8
> --
>
> Key: BEAM-3430
> URL: https://issues.apache.org/jira/browse/BEAM-3430
> Project: Beam
>  Issue Type: Sub-task
>  Components: website
>Reporter: Luke Cwik
>Priority: Major
> Fix For: 2.3.0
>
>
> I know of this one place but there are probably several others:
> https://beam.apache.org/get-started/quickstart-java/



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (BEAM-2680) Improve scalability of the Watch transform

[Eugene Kirpichov (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2680?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16339797#comment-16339797
 ] 

Eugene Kirpichov commented on BEAM-2680:


Note: as a workaround, normally a user should be able to "shard" the input of 
Watch (e.g. a filepattern) so that each individual poll result is smaller.

> Improve scalability of the Watch transform
> --
>
> Key: BEAM-2680
> URL: https://issues.apache.org/jira/browse/BEAM-2680
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Reporter: Eugene Kirpichov
>Assignee: Eugene Kirpichov
>Priority: Major
>
> [https://github.com/apache/beam/pull/3565] introduces the Watch transform 
> [http://s.apache.org/beam-watch-transform].
> The implementation leaves several scalability-related TODOs:
>  1) The state stores hashes and timestamps of outputs that have already been 
> output and should be omitted from future polls. We could garbage-collect this 
> state, e.g. dropping elements from "completed" and from addNewAsPending() if 
> their timestamp is more than X behind the watermark.
>  2) When a poll returns a huge number of elements, we don't necessarily have 
> to add all of them into state.pending - instead we could add only N oldest 
> elements and ignore others, relying on future poll rounds to provide them, in 
> order to avoid blowing up the state. Combined with garbage collection of 
> GrowthState.completed, this would make the transform scalable to very large 
> poll results.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (BEAM-2680) Improve scalability of the Watch transform

[Eugene Kirpichov (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2680?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Eugene Kirpichov updated BEAM-2680:
---
Description: 
[https://github.com/apache/beam/pull/3565] introduces the Watch transform 
[http://s.apache.org/beam-watch-transform].

The implementation leaves several scalability-related TODOs:
 1) The state stores hashes and timestamps of outputs that have already been 
output and should be omitted from future polls. We could garbage-collect this 
state, e.g. dropping elements from "completed" and from addNewAsPending() if 
their timestamp is more than X behind the watermark.
 2) When a poll returns a huge number of elements, we don't necessarily have to 
add all of them into state.pending - instead we could add only N oldest 
elements and ignore others, relying on future poll rounds to provide them, in 
order to avoid blowing up the state. Combined with garbage collection of 
GrowthState.completed, this would make the transform scalable to very large 
poll results.

  was:
https://github.com/apache/beam/pull/3565 introduces the Watch transform 
http://s.apache.org/beam-watch-transform.

The implementation leaves several scalability-related TODOs:
1) The state stores hashes and timestamps of outputs that have already been 
output and should be omitted from future polls. We could garbage-collect this 
state, e.g. dropping elements from "completed" and from addNewAsPending() if 
their timestamp is more than X behind the watermark.
2) When a poll returns a huge number of elements, we don't necessarily have to 
add all of them into state.pending - instead we could add only N elements and 
ignore others, relying on future poll rounds to provide them, in order to avoid 
blowing up the state. Combined with garbage collection of 
GrowthState.completed, this would make the transform scalable to very large 
poll results.


> Improve scalability of the Watch transform
> --
>
> Key: BEAM-2680
> URL: https://issues.apache.org/jira/browse/BEAM-2680
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Reporter: Eugene Kirpichov
>Assignee: Eugene Kirpichov
>Priority: Major
>
> [https://github.com/apache/beam/pull/3565] introduces the Watch transform 
> [http://s.apache.org/beam-watch-transform].
> The implementation leaves several scalability-related TODOs:
>  1) The state stores hashes and timestamps of outputs that have already been 
> output and should be omitted from future polls. We could garbage-collect this 
> state, e.g. dropping elements from "completed" and from addNewAsPending() if 
> their timestamp is more than X behind the watermark.
>  2) When a poll returns a huge number of elements, we don't necessarily have 
> to add all of them into state.pending - instead we could add only N oldest 
> elements and ignore others, relying on future poll rounds to provide them, in 
> order to avoid blowing up the state. Combined with garbage collection of 
> GrowthState.completed, this would make the transform scalable to very large 
> poll results.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: beam_PostCommit_Python_ValidatesContainer_Dataflow #10

[Apache Jenkins Server]

See 


--
[...truncated 91.77 KB...]
copying apache_beam/options/value_provider_test.py -> 
apache-beam-2.3.0.dev0/apache_beam/options
copying apache_beam/portability/__init__.py -> 
apache-beam-2.3.0.dev0/apache_beam/portability
copying apache_beam/portability/api/__init__.py -> 
apache-beam-2.3.0.dev0/apache_beam/portability/api
copying apache_beam/portability/api/beam_artifact_api_pb2.py -> 
apache-beam-2.3.0.dev0/apache_beam/portability/api
copying apache_beam/portability/api/beam_artifact_api_pb2_grpc.py -> 
apache-beam-2.3.0.dev0/apache_beam/portability/api
copying apache_beam/portability/api/beam_fn_api_pb2.py -> 
apache-beam-2.3.0.dev0/apache_beam/portability/api
copying apache_beam/portability/api/beam_fn_api_pb2_grpc.py -> 
apache-beam-2.3.0.dev0/apache_beam/portability/api
copying apache_beam/portability/api/beam_job_api_pb2.py -> 
apache-beam-2.3.0.dev0/apache_beam/portability/api
copying apache_beam/portability/api/beam_job_api_pb2_grpc.py -> 
apache-beam-2.3.0.dev0/apache_beam/portability/api
copying apache_beam/portability/api/beam_provision_api_pb2.py -> 
apache-beam-2.3.0.dev0/apache_beam/portability/api
copying apache_beam/portability/api/beam_provision_api_pb2_grpc.py -> 
apache-beam-2.3.0.dev0/apache_beam/portability/api
copying apache_beam/portability/api/beam_runner_api_pb2.py -> 
apache-beam-2.3.0.dev0/apache_beam/portability/api
copying apache_beam/portability/api/beam_runner_api_pb2_grpc.py -> 
apache-beam-2.3.0.dev0/apache_beam/portability/api
copying apache_beam/portability/api/endpoints_pb2.py -> 
apache-beam-2.3.0.dev0/apache_beam/portability/api
copying apache_beam/portability/api/endpoints_pb2_grpc.py -> 
apache-beam-2.3.0.dev0/apache_beam/portability/api
copying apache_beam/portability/api/standard_window_fns_pb2.py -> 
apache-beam-2.3.0.dev0/apache_beam/portability/api
copying apache_beam/portability/api/standard_window_fns_pb2_grpc.py -> 
apache-beam-2.3.0.dev0/apache_beam/portability/api
copying apache_beam/runners/__init__.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners
copying apache_beam/runners/common.pxd -> 
apache-beam-2.3.0.dev0/apache_beam/runners
copying apache_beam/runners/common.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners
copying apache_beam/runners/common_test.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners
copying apache_beam/runners/pipeline_context.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners
copying apache_beam/runners/pipeline_context_test.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners
copying apache_beam/runners/runner.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners
copying apache_beam/runners/runner_test.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners
copying apache_beam/runners/sdf_common.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners
copying apache_beam/runners/dataflow/__init__.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow
copying apache_beam/runners/dataflow/dataflow_metrics.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow
copying apache_beam/runners/dataflow/dataflow_metrics_test.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow
copying apache_beam/runners/dataflow/dataflow_runner.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow
copying apache_beam/runners/dataflow/dataflow_runner_test.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow
copying apache_beam/runners/dataflow/ptransform_overrides.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow
copying apache_beam/runners/dataflow/template_runner_test.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow
copying apache_beam/runners/dataflow/test_dataflow_runner.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow
copying apache_beam/runners/dataflow/internal/__init__.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/internal
copying apache_beam/runners/dataflow/internal/apiclient.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/internal
copying apache_beam/runners/dataflow/internal/apiclient_test.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/internal
copying apache_beam/runners/dataflow/internal/dependency.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/internal
copying apache_beam/runners/dataflow/internal/dependency_test.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/internal
copying apache_beam/runners/dataflow/internal/names.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/internal
copying apache_beam/runners/dataflow/internal/clients/__init__.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/internal/clients
copying apache_beam/runners/dataflow/internal/clients/dataflow/__init__.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/internal/clients/dataflow
copying 
apache_beam/runners/dataflow/internal/clients/dataflow/dataflow_v1b3_client.py 
-> 

[jira] [Resolved] (BEAM-3465) Remove the references in the website to Java 7

[JIRA]

 [ 
https://issues.apache.org/jira/browse/BEAM-3465?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jean-Baptiste Onofré resolved BEAM-3465.

   Resolution: Duplicate
Fix Version/s: Not applicable

> Remove the references in the website to Java 7
> --
>
> Key: BEAM-3465
> URL: https://issues.apache.org/jira/browse/BEAM-3465
> Project: Beam
>  Issue Type: Sub-task
>  Components: website
>Reporter: Ismaël Mejía
>Assignee: Jean-Baptiste Onofré
>Priority: Major
> Fix For: Not applicable
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (BEAM-3465) Remove the references in the website to Java 7

[JIRA]

 [ 
https://issues.apache.org/jira/browse/BEAM-3465?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jean-Baptiste Onofré updated BEAM-3465:
---
Fix Version/s: (was: 2.3.0)

> Remove the references in the website to Java 7
> --
>
> Key: BEAM-3465
> URL: https://issues.apache.org/jira/browse/BEAM-3465
> Project: Beam
>  Issue Type: Sub-task
>  Components: website
>Reporter: Ismaël Mejía
>Assignee: Jean-Baptiste Onofré
>Priority: Major
> Fix For: Not applicable
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (BEAM-3466) Remove Java 7 and any related task from Jenkins

[JIRA]

 [ 
https://issues.apache.org/jira/browse/BEAM-3466?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jean-Baptiste Onofré updated BEAM-3466:
---
Fix Version/s: 2.3.0

> Remove Java 7 and any related task from Jenkins
> ---
>
> Key: BEAM-3466
> URL: https://issues.apache.org/jira/browse/BEAM-3466
> Project: Beam
>  Issue Type: Sub-task
>  Components: build-system
>Reporter: Ismaël Mejía
>Assignee: Jean-Baptiste Onofré
>Priority: Major
> Fix For: 2.3.0
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (BEAM-3466) Remove Java 7 and any related task from Jenkins

[JIRA]

 [ 
https://issues.apache.org/jira/browse/BEAM-3466?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jean-Baptiste Onofré updated BEAM-3466:
---
Fix Version/s: (was: 2.3.0)

> Remove Java 7 and any related task from Jenkins
> ---
>
> Key: BEAM-3466
> URL: https://issues.apache.org/jira/browse/BEAM-3466
> Project: Beam
>  Issue Type: Sub-task
>  Components: build-system
>Reporter: Ismaël Mejía
>Assignee: Jean-Baptiste Onofré
>Priority: Major
> Fix For: 2.3.0
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (BEAM-3466) Remove Java 7 and any related task from Jenkins

[JIRA]

[ 
https://issues.apache.org/jira/browse/BEAM-3466?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16339786#comment-16339786
 ] 

Jean-Baptiste Onofré commented on BEAM-3466:


Just open a PR about this one.

> Remove Java 7 and any related task from Jenkins
> ---
>
> Key: BEAM-3466
> URL: https://issues.apache.org/jira/browse/BEAM-3466
> Project: Beam
>  Issue Type: Sub-task
>  Components: build-system
>Reporter: Ismaël Mejía
>Assignee: Jean-Baptiste Onofré
>Priority: Major
> Fix For: 2.3.0
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (BEAM-3465) Remove the references in the website to Java 7

[JIRA]

 [ 
https://issues.apache.org/jira/browse/BEAM-3465?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jean-Baptiste Onofré reassigned BEAM-3465:
--

Assignee: Jean-Baptiste Onofré

> Remove the references in the website to Java 7
> --
>
> Key: BEAM-3465
> URL: https://issues.apache.org/jira/browse/BEAM-3465
> Project: Beam
>  Issue Type: Sub-task
>  Components: website
>Reporter: Ismaël Mejía
>Assignee: Jean-Baptiste Onofré
>Priority: Major
> Fix For: 2.3.0
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (BEAM-3465) Remove the references in the website to Java 7

[JIRA]

[ 
https://issues.apache.org/jira/browse/BEAM-3465?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16339787#comment-16339787
 ] 

Jean-Baptiste Onofré commented on BEAM-3465:


Not release blocker anyway.

> Remove the references in the website to Java 7
> --
>
> Key: BEAM-3465
> URL: https://issues.apache.org/jira/browse/BEAM-3465
> Project: Beam
>  Issue Type: Sub-task
>  Components: website
>Reporter: Ismaël Mejía
>Assignee: Jean-Baptiste Onofré
>Priority: Major
> Fix For: 2.3.0
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (BEAM-3466) Remove Java 7 and any related task from Jenkins

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-3466?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16339785#comment-16339785
 ] 

ASF GitHub Bot commented on BEAM-3466:
--

jbonofre opened a new pull request #4489: [BEAM-3466] Remove JDK 7 references 
in Jenkins
URL: https://github.com/apache/beam/pull/4489
 
 
   Follow this checklist to help us incorporate your contribution quickly and 
easily:
   
- [X] Make sure there is a [JIRA 
issue](https://issues.apache.org/jira/projects/BEAM/issues/) filed for the 
change (usually before you start working on it).  Trivial changes like typos do 
not require a JIRA issue.  Your pull request should address just this issue, 
without pulling in other changes.
- [X] Each commit in the pull request should have a meaningful subject line 
and body.
- [X] Format the pull request title like `[BEAM-XXX] Fixes bug in 
ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA 
issue.
- [X] Write a pull request description that is detailed enough to 
understand what the pull request does, how, and why.
- [X] Run `mvn clean verify` to make sure basic checks pass. A more 
thorough check will be performed on your pull request automatically.
- [X] If this contribution is large, please file an Apache [Individual 
Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
   
   ---
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Remove Java 7 and any related task from Jenkins
> ---
>
> Key: BEAM-3466
> URL: https://issues.apache.org/jira/browse/BEAM-3466
> Project: Beam
>  Issue Type: Sub-task
>  Components: build-system
>Reporter: Ismaël Mejía
>Assignee: Jean-Baptiste Onofré
>Priority: Major
> Fix For: 2.3.0
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (BEAM-3243) multiple anonymous DoFn lead to conflicting names

[JIRA]

 [ 
https://issues.apache.org/jira/browse/BEAM-3243?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jean-Baptiste Onofré resolved BEAM-3243.

Resolution: Fixed

This has been fixed with PR #4185

> multiple anonymous DoFn lead to conflicting names
> -
>
> Key: BEAM-3243
> URL: https://issues.apache.org/jira/browse/BEAM-3243
> Project: Beam
>  Issue Type: Task
>  Components: sdk-java-core
>Reporter: Romain Manni-Bucau
>Assignee: Romain Manni-Bucau
>Priority: Major
> Fix For: 2.3.0
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (BEAM-3533) Replace hard-coded UTF-8 Strings

[JIRA]

[ 
https://issues.apache.org/jira/browse/BEAM-3533?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16339765#comment-16339765
 ] 

Jean-Baptiste Onofré commented on BEAM-3533:


[~lcwik] PR #4486 superseeded #4485. [~coheigea] should have updated the #4485 
PR instead of creating a new one. Next time I will guide him ;)

> Replace hard-coded UTF-8 Strings
> 
>
> Key: BEAM-3533
> URL: https://issues.apache.org/jira/browse/BEAM-3533
> Project: Beam
>  Issue Type: Improvement
>  Components: beam-model
>Reporter: Colm O hEigeartaigh
>Assignee: Jean-Baptiste Onofré
>Priority: Trivial
> Fix For: 2.3.0
>
>
> This task is to replace hard-coded UTF-8 Strings in the code with 
> StandardCharsets.UTF_8 instead.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (BEAM-3533) Replace hard-coded UTF-8 Strings

[JIRA]

 [ 
https://issues.apache.org/jira/browse/BEAM-3533?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jean-Baptiste Onofré resolved BEAM-3533.

Resolution: Fixed

> Replace hard-coded UTF-8 Strings
> 
>
> Key: BEAM-3533
> URL: https://issues.apache.org/jira/browse/BEAM-3533
> Project: Beam
>  Issue Type: Improvement
>  Components: beam-model
>Reporter: Colm O hEigeartaigh
>Assignee: Jean-Baptiste Onofré
>Priority: Trivial
> Fix For: 2.3.0
>
>
> This task is to replace hard-coded UTF-8 Strings in the code with 
> StandardCharsets.UTF_8 instead.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: beam_PostCommit_Python_Verify #4055

[Apache Jenkins Server]

See 


Changes:

[coheigea] BEAM-3533 - Replace hard-coded UTF-8 Strings

--
[...truncated 1.40 MB...]
copying apache_beam/runners/experimental/__init__.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/experimental
copying apache_beam/runners/experimental/python_rpc_direct/__init__.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/experimental/python_rpc_direct
copying 
apache_beam/runners/experimental/python_rpc_direct/python_rpc_direct_runner.py 
-> apache-beam-2.3.0.dev0/apache_beam/runners/experimental/python_rpc_direct
copying apache_beam/runners/experimental/python_rpc_direct/server.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/experimental/python_rpc_direct
copying apache_beam/runners/job/__init__.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/job
copying apache_beam/runners/job/manager.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/job
copying apache_beam/runners/job/utils.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/job
copying apache_beam/runners/portability/__init__.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/portability
copying apache_beam/runners/portability/fn_api_runner.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/portability
copying apache_beam/runners/portability/fn_api_runner_test.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/portability
copying apache_beam/runners/portability/maptask_executor_runner.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/portability
copying apache_beam/runners/portability/maptask_executor_runner_test.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/portability
copying apache_beam/runners/portability/universal_local_runner.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/portability
copying apache_beam/runners/portability/universal_local_runner_main.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/portability
copying apache_beam/runners/portability/universal_local_runner_test.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/portability
copying apache_beam/runners/test/__init__.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/test
copying apache_beam/runners/worker/__init__.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/worker
copying apache_beam/runners/worker/bundle_processor.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/worker
copying apache_beam/runners/worker/data_plane.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/worker
copying apache_beam/runners/worker/data_plane_test.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/worker
copying apache_beam/runners/worker/log_handler.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/worker
copying apache_beam/runners/worker/log_handler_test.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/worker
copying apache_beam/runners/worker/logger.pxd -> 
apache-beam-2.3.0.dev0/apache_beam/runners/worker
copying apache_beam/runners/worker/logger.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/worker
copying apache_beam/runners/worker/logger_test.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/worker
copying apache_beam/runners/worker/opcounters.pxd -> 
apache-beam-2.3.0.dev0/apache_beam/runners/worker
copying apache_beam/runners/worker/opcounters.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/worker
copying apache_beam/runners/worker/opcounters_test.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/worker
copying apache_beam/runners/worker/operation_specs.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/worker
copying apache_beam/runners/worker/operations.pxd -> 
apache-beam-2.3.0.dev0/apache_beam/runners/worker
copying apache_beam/runners/worker/operations.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/worker
copying apache_beam/runners/worker/sdk_worker.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/worker
copying apache_beam/runners/worker/sdk_worker_main.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/worker
copying apache_beam/runners/worker/sdk_worker_main_test.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/worker
copying apache_beam/runners/worker/sdk_worker_test.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/worker
copying apache_beam/runners/worker/sideinputs.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/worker
copying apache_beam/runners/worker/sideinputs_test.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/worker
copying apache_beam/runners/worker/statesampler.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/worker
copying apache_beam/runners/worker/statesampler_fast.pyx -> 
apache-beam-2.3.0.dev0/apache_beam/runners/worker
copying apache_beam/runners/worker/statesampler_slow.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/worker
copying apache_beam/runners/worker/statesampler_test.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/worker
copying apache_beam/testing/__init__.py -> 
apache-beam-2.3.0.dev0/apache_beam/testing
copying apache_beam/testing/pipeline_verifiers.py -> 

Build failed in Jenkins: beam_PostCommit_Python_ValidatesContainer_Dataflow #9

[Apache Jenkins Server]

See 


Changes:

[robert] Add CONTRIBUTING.md

[iemejia] [BEAM-3432] Remove hadoop/jdk1.8-tests module

[lcwik] [BEAM-3160] Fix issue where we would choose which coder to use

[lcwik] Initial post-release snapshot test

[lcwik] Make the snapshot and url parameters to the job to allow installing a

[lcwik] Make the snapshot and url parameters to the job to allow installing a

[lcwik] Rename TestHelper to TestScripts, it() to intent()

[chamikara] [BEAM-3060] Support for Perfkit execution of file-based-io-tests on 
HDFS

[jbonofre] [BEAM-3428] Merge Java8 examples in "main" Java examples

[jbonofre] [BEAM-3428] Replace MinimalWordCount with Java8 one

[echauchot] [BEAM-3534] Add a spark validates runner test for metrics sink in

[coheigea] BEAM-3533 - Replace hard-coded UTF-8 Strings

--
Started by GitHub push by jbonofre
[EnvInject] - Loading node environment variables.
Building remotely on beam1 (beam) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* 
 > +refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/*
 > git rev-parse origin/master^{commit} # timeout=10
Checking out Revision e28c4da27f781fa38edebfe3140a71b921aaa242 (origin/master)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f e28c4da27f781fa38edebfe3140a71b921aaa242
Commit message: "Merge pull request #4486 from coheigea/BEAM-3533"
 > git rev-list 7f99628784c53c1b28e38bb4bfd9448c94021011 # timeout=10
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
[EnvInject] - Executing scripts and injecting environment variables after the 
SCM step.
[EnvInject] - Injecting as environment variables the properties content 
SPARK_LOCAL_IP=127.0.0.1

[EnvInject] - Variables injected successfully.
[beam_PostCommit_Python_ValidatesContainer_Dataflow] $ /bin/bash -xe 
/tmp/jenkins5702224648851591361.sh
+ cd src
+ bash sdks/python/container/run_validatescontainer.sh
bash: sdks/python/container/run_validatescontainer.sh: No such file or directory
Build step 'Execute shell' marked build as failure
Not sending mail to unregistered user rob...@frantil.com

Jenkins build is still unstable: beam_PostCommit_Java_ValidatesRunner_Spark #3992

[Apache Jenkins Server]

See 

[jira] [Commented] (BEAM-3243) multiple anonymous DoFn lead to conflicting names

[Kenneth Knowles (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-3243?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16339689#comment-16339689
 ] 

Kenneth Knowles commented on BEAM-3243:
---

I see the pull requests associated with this are merged or closed. Is it 
resolved?

> multiple anonymous DoFn lead to conflicting names
> -
>
> Key: BEAM-3243
> URL: https://issues.apache.org/jira/browse/BEAM-3243
> Project: Beam
>  Issue Type: Task
>  Components: sdk-java-core
>Reporter: Romain Manni-Bucau
>Assignee: Romain Manni-Bucau
>Priority: Major
> Fix For: 2.3.0
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (BEAM-3286) Go SDK support for portable side input

[Kenneth Knowles (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3286?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kenneth Knowles updated BEAM-3286:
--
Fix Version/s: (was: 2.3.0)

> Go SDK support for portable side input
> --
>
> Key: BEAM-3286
> URL: https://issues.apache.org/jira/browse/BEAM-3286
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-go
>Reporter: Henning Rohde
>Assignee: Henning Rohde
>Priority: Major
>  Labels: portability
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (BEAM-3453) Allow usage of public Google PubSub topics in Python DirectRunner

[Kenneth Knowles (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3453?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kenneth Knowles updated BEAM-3453:
--
Issue Type: New Feature  (was: Bug)

> Allow usage of public Google PubSub topics in Python DirectRunner
> -
>
> Key: BEAM-3453
> URL: https://issues.apache.org/jira/browse/BEAM-3453
> Project: Beam
>  Issue Type: New Feature
>  Components: sdk-py-core
>Affects Versions: 2.2.0
>Reporter: Charles Chen
>Assignee: Charles Chen
>Priority: Major
>
> Currently, the Beam Python DirectRunner does not allow the usage of data from 
> public Google Cloud PubSub topics.  We should allow this functionality so 
> that users can more easily test Beam Python's streaming functionality.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (BEAM-3453) Allow usage of public Google PubSub topics in Python DirectRunner

[Kenneth Knowles (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-3453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16339686#comment-16339686
 ] 

Kenneth Knowles commented on BEAM-3453:
---

This seems like a feature, so it should just acquire a Fix Version when it is 
completed, to generate a changelog.

> Allow usage of public Google PubSub topics in Python DirectRunner
> -
>
> Key: BEAM-3453
> URL: https://issues.apache.org/jira/browse/BEAM-3453
> Project: Beam
>  Issue Type: New Feature
>  Components: sdk-py-core
>Affects Versions: 2.2.0
>Reporter: Charles Chen
>Assignee: Charles Chen
>Priority: Major
>
> Currently, the Beam Python DirectRunner does not allow the usage of data from 
> public Google Cloud PubSub topics.  We should allow this functionality so 
> that users can more easily test Beam Python's streaming functionality.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (BEAM-3453) Allow usage of public Google PubSub topics in Python DirectRunner

[Kenneth Knowles (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3453?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kenneth Knowles updated BEAM-3453:
--
Fix Version/s: (was: 2.3.0)

> Allow usage of public Google PubSub topics in Python DirectRunner
> -
>
> Key: BEAM-3453
> URL: https://issues.apache.org/jira/browse/BEAM-3453
> Project: Beam
>  Issue Type: New Feature
>  Components: sdk-py-core
>Affects Versions: 2.2.0
>Reporter: Charles Chen
>Assignee: Charles Chen
>Priority: Major
>
> Currently, the Beam Python DirectRunner does not allow the usage of data from 
> public Google Cloud PubSub topics.  We should allow this functionality so 
> that users can more easily test Beam Python's streaming functionality.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (BEAM-3466) Remove Java 7 and any related task from Jenkins

[Kenneth Knowles (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-3466?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16339684#comment-16339684
 ] 

Kenneth Knowles commented on BEAM-3466:
---

This doesn't block release, right? Well, actually, since the Jenkins jobs from 
{{master}} will be the ones that validate the release branch, we should perhaps 
do as much work as we can before we cut the release branch so that we don't 
have the sync problems we had last time.

> Remove Java 7 and any related task from Jenkins
> ---
>
> Key: BEAM-3466
> URL: https://issues.apache.org/jira/browse/BEAM-3466
> Project: Beam
>  Issue Type: Sub-task
>  Components: build-system
>Reporter: Ismaël Mejía
>Assignee: Jean-Baptiste Onofré
>Priority: Major
> Fix For: 2.3.0
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (BEAM-3509) PARTITION BY in Beam SQL In Select Command

[Kenneth Knowles (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-3509?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16339676#comment-16339676
 ] 

Kenneth Knowles commented on BEAM-3509:
---

Does this have any work in progress? I think we should remove the Fix Version 
and it can land when it is done.

> PARTITION BY in Beam SQL In Select Command
> --
>
> Key: BEAM-3509
> URL: https://issues.apache.org/jira/browse/BEAM-3509
> Project: Beam
>  Issue Type: New Feature
>  Components: dsl-sql
>Affects Versions: 2.2.0
>Reporter: Kishan Kumar
>Assignee: Xu Mingmin
>Priority: Major
>  Labels: performance
>
> Partition By Option Will Be Very Help Full for DataFlow Developer To Migrate 
> Query and Do Transformation on That because of Many *Netezza Query and Oracle 
> Query* Consists Of Partition By Which Makes SQL Query More Efficient. *The 
> alternative is Making Joins And Filtering It Can Be Done But It Makes Code 
> Unreadable And Performance Become bad for DataFlow Job.*
> Examples: SELECT MIN(COLUMN) OVER (PARTITION BY COLUMN NAME) FROM TABLENAME



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (BEAM-3509) PARTITION BY in Beam SQL In Select Command

[Kenneth Knowles (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3509?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kenneth Knowles updated BEAM-3509:
--
Fix Version/s: (was: 2.3.0)

> PARTITION BY in Beam SQL In Select Command
> --
>
> Key: BEAM-3509
> URL: https://issues.apache.org/jira/browse/BEAM-3509
> Project: Beam
>  Issue Type: New Feature
>  Components: dsl-sql
>Affects Versions: 2.2.0
>Reporter: Kishan Kumar
>Assignee: Xu Mingmin
>Priority: Major
>  Labels: performance
>
> Partition By Option Will Be Very Help Full for DataFlow Developer To Migrate 
> Query and Do Transformation on That because of Many *Netezza Query and Oracle 
> Query* Consists Of Partition By Which Makes SQL Query More Efficient. *The 
> alternative is Making Joins And Filtering It Can Be Done But It Makes Code 
> Unreadable And Performance Become bad for DataFlow Job.*
> Examples: SELECT MIN(COLUMN) OVER (PARTITION BY COLUMN NAME) FROM TABLENAME



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (BEAM-3514) Use portable WindowIntoPayload in DataflowRunner

[Kenneth Knowles (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3514?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kenneth Knowles updated BEAM-3514:
--
Fix Version/s: (was: 2.3.0)

> Use portable WindowIntoPayload in DataflowRunner
> 
>
> Key: BEAM-3514
> URL: https://issues.apache.org/jira/browse/BEAM-3514
> Project: Beam
>  Issue Type: Sub-task
>  Components: runner-dataflow
>Reporter: Kenneth Knowles
>Priority: Major
>  Labels: portability
>
> The Java-specific blobs transmitted to Dataflow need more context, in the 
> form of portability framework protos.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (BEAM-3513) Use portable CombinePayload in Java DataflowRunner

[Kenneth Knowles (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3513?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kenneth Knowles updated BEAM-3513:
--
Fix Version/s: (was: 2.3.0)

> Use portable CombinePayload in Java DataflowRunner
> --
>
> Key: BEAM-3513
> URL: https://issues.apache.org/jira/browse/BEAM-3513
> Project: Beam
>  Issue Type: Sub-task
>  Components: runner-dataflow
>Reporter: Kenneth Knowles
>Assignee: Daniel Oliveira
>Priority: Major
>  Labels: portability
>
> The Java-specific blobs transmitted to Dataflow need more context, in the 
> form of portability framework protos.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (BEAM-2345) Version configuration of plugins / dependencies in root pom.xml is inconsistent

[Kenneth Knowles (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2345?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16339672#comment-16339672
 ] 

Kenneth Knowles commented on BEAM-2345:
---

I think this might be partially addressed, but anyhow we shouldn't invest too 
much in refactors of poms since we are moving to gradle.

> Version configuration of plugins / dependencies in root pom.xml is 
> inconsistent
> ---
>
> Key: BEAM-2345
> URL: https://issues.apache.org/jira/browse/BEAM-2345
> Project: Beam
>  Issue Type: Bug
>  Components: build-system
>Reporter: Jason Kuster
>Priority: Minor
> Fix For: Not applicable
>
>
> Versioning in root pom.xml in some places is controlled by the properties 
> section, sometimes is just inline. Move all versioning of plugins / 
> dependencies to properties section.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (BEAM-2345) Version configuration of plugins / dependencies in root pom.xml is inconsistent

[Kenneth Knowles (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2345?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kenneth Knowles resolved BEAM-2345.
---
   Resolution: Won't Fix
Fix Version/s: Not applicable

> Version configuration of plugins / dependencies in root pom.xml is 
> inconsistent
> ---
>
> Key: BEAM-2345
> URL: https://issues.apache.org/jira/browse/BEAM-2345
> Project: Beam
>  Issue Type: Bug
>  Components: build-system
>Reporter: Jason Kuster
>Priority: Minor
> Fix For: Not applicable
>
>
> Versioning in root pom.xml in some places is controlled by the properties 
> section, sometimes is just inline. Move all versioning of plugins / 
> dependencies to properties section.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (BEAM-2345) Version configuration of plugins / dependencies in root pom.xml is inconsistent

[Kenneth Knowles (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2345?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kenneth Knowles updated BEAM-2345:
--
Fix Version/s: (was: 2.3.0)

> Version configuration of plugins / dependencies in root pom.xml is 
> inconsistent
> ---
>
> Key: BEAM-2345
> URL: https://issues.apache.org/jira/browse/BEAM-2345
> Project: Beam
>  Issue Type: Bug
>  Components: build-system
>Reporter: Jason Kuster
>Priority: Minor
>
> Versioning in root pom.xml in some places is controlled by the properties 
> section, sometimes is just inline. Move all versioning of plugins / 
> dependencies to properties section.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (BEAM-3392) Verify default container image names in Python SDK

[Kenneth Knowles (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-3392?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16339668#comment-16339668
 ] 

Kenneth Knowles commented on BEAM-3392:
---

Do I understand that this is a manual step you want a Dataflow maintainer to 
perform when Beam makes a 2.3.0 release?

> Verify default container image names in Python SDK
> --
>
> Key: BEAM-3392
> URL: https://issues.apache.org/jira/browse/BEAM-3392
> Project: Beam
>  Issue Type: Test
>  Components: runner-dataflow
>Affects Versions: 2.3.0
>Reporter: Valentyn Tymofieiev
>Assignee: Thomas Groh
>Priority: Blocker
> Fix For: 2.3.0
>
>
> Verify naming pattern for python containers in 2.3.0 release:
> - default legacy worker harness container image: 
> dataflow.gcr.io/v1beta3/python:2.3.0
> - pinned runner harness container image: dataflow.gcr.io/v1beta3/harness:2.3.0
> - default SDK harness container name (portable pipelines): name may change, 
> exact pattern TBD.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (BEAM-3332) AfterProcessingTimer trigger not firing if invoked exactly on time

[Kenneth Knowles (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3332?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kenneth Knowles resolved BEAM-3332.
---
   Resolution: Not A Bug
Fix Version/s: Not applicable

I am going to close as "Not a Bug" because I think that it works as it is 
supposed to, and the problem is the underlying timer delivery (the timers show 
up too early and have no effect). We could warn or throw in some cases if a 
timer is delivered before it should, but so far we wanted to be very flexible 
and resilient to that.

> AfterProcessingTimer trigger not firing if invoked exactly on time
> --
>
> Key: BEAM-3332
> URL: https://issues.apache.org/jira/browse/BEAM-3332
> Project: Beam
>  Issue Type: Bug
>  Components: runner-core
>Affects Versions: 2.0.0, 2.1.0, 2.2.0
>Reporter: Shen Li
>Assignee: Shen Li
>Priority: Major
> Fix For: Not applicable
>
>
> I occasionally run into an issue that the processing time trigger is invoked 
> on time, but the TriggerStateMachienRunner#shouldFire() returns false. After 
> comparing time instances, I found that this issue occurs when the trigger is 
> invoked exactly on time. It is because the 
> AfterDelayFromFirstElementStateMachine does the following:
> {quote}return delayedUntil != null
> && getCurrentTime(context) != null
> && getCurrentTime(context).isAfter(delayedUntil);{quote}
> which only returns true when the current processing time is AFTER (exclude 
> equals) delayUntil. Should it actually be 
> !getCurrentTime(context).isBefore(delayedUntil) ?



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (BEAM-3392) Verify default container image names in Python SDK

[Kenneth Knowles (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3392?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kenneth Knowles updated BEAM-3392:
--
Priority: Blocker  (was: Minor)

> Verify default container image names in Python SDK
> --
>
> Key: BEAM-3392
> URL: https://issues.apache.org/jira/browse/BEAM-3392
> Project: Beam
>  Issue Type: Test
>  Components: runner-dataflow
>Affects Versions: 2.3.0
>Reporter: Valentyn Tymofieiev
>Assignee: Thomas Groh
>Priority: Blocker
> Fix For: 2.3.0
>
>
> Verify naming pattern for python containers in 2.3.0 release:
> - default legacy worker harness container image: 
> dataflow.gcr.io/v1beta3/python:2.3.0
> - pinned runner harness container image: dataflow.gcr.io/v1beta3/harness:2.3.0
> - default SDK harness container name (portable pipelines): name may change, 
> exact pattern TBD.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)