Jenkins build became unstable: beam_PostCommit_Java_MavenInstall #5305

[Apache Jenkins Server]

See 

[jira] [Commented] (BEAM-3242) Gearpump runner does not respect ParDo's lifecycle on case of exceptions

[Manu Zhang (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-3242?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16265606#comment-16265606
 ] 

Manu Zhang commented on BEAM-3242:
--

Hi Ismael, thanks for reporting. Could you elaborate on what should be 
respected in case of exceptions ?

> Gearpump runner does not respect ParDo's lifecycle on case of exceptions
> 
>
> Key: BEAM-3242
> URL: https://issues.apache.org/jira/browse/BEAM-3242
> Project: Beam
>  Issue Type: Bug
>  Components: runner-gearpump
>Affects Versions: 2.2.0
>Reporter: Ismaël Mejía
>Assignee: Manu Zhang
>Priority: Minor
>
> The lifecycle of the DoFn is not respected in case of exception in any of the 
> lifecycle methods after setup.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Jenkins build became unstable: beam_PostCommit_Java_ValidatesRunner_Flink #4344

[Apache Jenkins Server]

See 

Build failed in Jenkins: beam_PerformanceTests_Spark #1045

[Apache Jenkins Server]

See 


--
Started by timer
[EnvInject] - Loading node environment variables.
Building remotely on beam2 (beam) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* 
 > +refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/*
 > git rev-parse origin/master^{commit} # timeout=10
Checking out Revision d6345e52e6e8b319456cbad6ce11b05399815930 (origin/master)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f d6345e52e6e8b319456cbad6ce11b05399815930
Commit message: "Update scm to use gitbox.apache.org"
 > git rev-list d6345e52e6e8b319456cbad6ce11b05399815930 # timeout=10
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
[EnvInject] - Executing scripts and injecting environment variables after the 
SCM step.
[EnvInject] - Injecting as environment variables the properties content 
SPARK_LOCAL_IP=127.0.0.1

[EnvInject] - Variables injected successfully.
[beam_PerformanceTests_Spark] $ /bin/bash -xe /tmp/jenkins9146070859787339578.sh
+ rm -rf PerfKitBenchmarker
[beam_PerformanceTests_Spark] $ /bin/bash -xe /tmp/jenkins786830982489293720.sh
+ git clone https://github.com/GoogleCloudPlatform/PerfKitBenchmarker.git
Cloning into 'PerfKitBenchmarker'...
[beam_PerformanceTests_Spark] $ /bin/bash -xe /tmp/jenkins886059065617717426.sh
+ pip install --user -r PerfKitBenchmarker/requirements.txt
Requirement already satisfied: absl-py in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 14))
Requirement already satisfied: jinja2>=2.7 in 
/usr/local/lib/python2.7/dist-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 15))
Requirement already satisfied: setuptools in /usr/lib/python2.7/dist-packages 
(from -r PerfKitBenchmarker/requirements.txt (line 16))
Requirement already satisfied: colorlog[windows]==2.6.0 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 17))
Requirement already satisfied: blinker>=1.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 18))
Requirement already satisfied: futures>=3.0.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 19))
Requirement already satisfied: PyYAML==3.12 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 20))
Requirement already satisfied: pint>=0.7 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 21))
Requirement already satisfied: numpy in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 22))
Requirement already satisfied: functools32 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 23))
Requirement already satisfied: contextlib2>=0.5.1 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 24))
Requirement already satisfied: pywinrm in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 25))
Requirement already satisfied: six in 
/home/jenkins/.local/lib/python2.7/site-packages (from absl-py->-r 
PerfKitBenchmarker/requirements.txt (line 14))
Requirement already satisfied: MarkupSafe in 
/usr/local/lib/python2.7/dist-packages (from jinja2>=2.7->-r 
PerfKitBenchmarker/requirements.txt (line 15))
Requirement already satisfied: colorama; extra == "windows" in 
/usr/lib/python2.7/dist-packages (from colorlog[windows]==2.6.0->-r 
PerfKitBenchmarker/requirements.txt (line 17))
Requirement already satisfied: xmltodict in 
/home/jenkins/.local/lib/python2.7/site-packages (from pywinrm->-r 
PerfKitBenchmarker/requirements.txt (line 25))
Requirement already satisfied: requests-ntlm>=0.3.0 in 
/home/jenkins/.local/lib/python2.7/site-packages (from pywinrm->-r 
PerfKitBenchmarker/requirements.txt (line 25))
Requirement already satisfied: requests>=2.9.1 in 
/home/jenkins/.local/lib/python2.7/site-packages (from pywinrm->-r 
PerfKitBenchmarker/requirements.txt (line 25))
Requirement already satisfied: ntlm-auth>=1.0.2 in 
/home/jenkins/.local/lib/python2.7/site-packages (from 
requests-ntlm>=0.3.0->pywinrm->-r PerfKitBenchmarker/requirements.txt (line 25))
Requirement already satisfied: cryptography>=1.3 in 

Build failed in Jenkins: beam_PerformanceTests_Python #598

[Apache Jenkins Server]

See 


--
Started by timer
[EnvInject] - Loading node environment variables.
Building remotely on beam7 (beam) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* 
 > +refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/*
 > git rev-parse origin/master^{commit} # timeout=10
Checking out Revision d6345e52e6e8b319456cbad6ce11b05399815930 (origin/master)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f d6345e52e6e8b319456cbad6ce11b05399815930
Commit message: "Update scm to use gitbox.apache.org"
 > git rev-list d6345e52e6e8b319456cbad6ce11b05399815930 # timeout=10
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
[EnvInject] - Executing scripts and injecting environment variables after the 
SCM step.
[EnvInject] - Injecting as environment variables the properties content 
SPARK_LOCAL_IP=127.0.0.1

[EnvInject] - Variables injected successfully.
[beam_PerformanceTests_Python] $ /bin/bash -xe /tmp/jenkins849460650283196767.sh
+ rm -rf PerfKitBenchmarker
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins7806951547103960598.sh
+ git clone https://github.com/GoogleCloudPlatform/PerfKitBenchmarker.git
Cloning into 'PerfKitBenchmarker'...
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins6036187315247093635.sh
+ pip install --user -r PerfKitBenchmarker/requirements.txt
Requirement already satisfied: absl-py in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 14))
Requirement already satisfied: jinja2>=2.7 in 
/usr/local/lib/python2.7/dist-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 15))
Requirement already satisfied: setuptools in /usr/lib/python2.7/dist-packages 
(from -r PerfKitBenchmarker/requirements.txt (line 16))
Requirement already satisfied: colorlog[windows]==2.6.0 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 17))
Requirement already satisfied: blinker>=1.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 18))
Requirement already satisfied: futures>=3.0.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 19))
Requirement already satisfied: PyYAML==3.12 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 20))
Requirement already satisfied: pint>=0.7 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 21))
Requirement already satisfied: numpy in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 22))
Requirement already satisfied: functools32 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 23))
Requirement already satisfied: contextlib2>=0.5.1 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 24))
Requirement already satisfied: pywinrm in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 25))
Requirement already satisfied: six in 
/home/jenkins/.local/lib/python2.7/site-packages (from absl-py->-r 
PerfKitBenchmarker/requirements.txt (line 14))
Requirement already satisfied: MarkupSafe>=0.23 in 
/usr/local/lib/python2.7/dist-packages (from jinja2>=2.7->-r 
PerfKitBenchmarker/requirements.txt (line 15))
Requirement already satisfied: colorama; extra == "windows" in 
/usr/lib/python2.7/dist-packages (from colorlog[windows]==2.6.0->-r 
PerfKitBenchmarker/requirements.txt (line 17))
Requirement already satisfied: xmltodict in 
/home/jenkins/.local/lib/python2.7/site-packages (from pywinrm->-r 
PerfKitBenchmarker/requirements.txt (line 25))
Requirement already satisfied: requests-ntlm>=0.3.0 in 
/home/jenkins/.local/lib/python2.7/site-packages (from pywinrm->-r 
PerfKitBenchmarker/requirements.txt (line 25))
Requirement already satisfied: requests>=2.9.1 in 
/home/jenkins/.local/lib/python2.7/site-packages (from pywinrm->-r 
PerfKitBenchmarker/requirements.txt (line 25))
Requirement already satisfied: ntlm-auth>=1.0.2 in 
/home/jenkins/.local/lib/python2.7/site-packages (from 
requests-ntlm>=0.3.0->pywinrm->-r PerfKitBenchmarker/requirements.txt (line 25))
Requirement already satisfied: 

Build failed in Jenkins: beam_PostCommit_Python_Verify #3609

[Apache Jenkins Server]

See 


--
Started by timer
[EnvInject] - Loading node environment variables.
Building remotely on beam3 (beam) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* 
 > +refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/*
 > git rev-parse origin/master^{commit} # timeout=10
Checking out Revision d6345e52e6e8b319456cbad6ce11b05399815930 (origin/master)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f d6345e52e6e8b319456cbad6ce11b05399815930
Commit message: "Update scm to use gitbox.apache.org"
 > git rev-list d6345e52e6e8b319456cbad6ce11b05399815930 # timeout=10
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
[EnvInject] - Executing scripts and injecting environment variables after the 
SCM step.
[EnvInject] - Injecting as environment variables the properties content 
SPARK_LOCAL_IP=127.0.0.1

[EnvInject] - Variables injected successfully.
[beam_PostCommit_Python_Verify] $ /bin/bash -xe 
/tmp/jenkins7983257134653980367.sh
+ bash sdks/python/run_postcommit.sh
bash: sdks/python/run_postcommit.sh: No such file or directory
Build step 'Execute shell' marked build as failure
Not sending mail to unregistered user ajam...@ajamato2016.sea.corp.google.com
Not sending mail to unregistered user c...@google.com
Not sending mail to unregistered user hero...@google.com
Not sending mail to unregistered user git...@nakomis.com
Not sending mail to unregistered user valen...@google.com
Not sending mail to unregistered user 
ke...@kedin-macbookpro.roam.corp.google.com
Not sending mail to unregistered user xumingmi...@gmail.com
Not sending mail to unregistered user k...@google.com
Not sending mail to unregistered user kirpic...@google.com
Not sending mail to unregistered user ekirpic...@gmail.com
ERROR: Unable to send to address: echauc...@gmail.com+jbono...@apache.org
javax.mail.internet.AddressException: Domain contains illegal character in 
string ``echauc...@gmail.com+jbono...@apache.org''
at 
javax.mail.internet.InternetAddress.checkAddress(InternetAddress.java:1269)
at javax.mail.internet.InternetAddress.parse(InternetAddress.java:1091)
at javax.mail.internet.InternetAddress.parse(InternetAddress.java:633)
at javax.mail.internet.InternetAddress.(InternetAddress.java:111)
at hudson.tasks.Mailer.stringToAddress(Mailer.java:206)
at 
jenkins.plugins.mailer.tasks.MimeMessageBuilder.toNormalizedAddress(MimeMessageBuilder.java:309)
at 
jenkins.plugins.mailer.tasks.MimeMessageBuilder.addRecipients(MimeMessageBuilder.java:168)
at 
jenkins.plugins.mailer.tasks.MimeMessageBuilder.addRecipients(MimeMessageBuilder.java:160)
at hudson.tasks.MailSender.createEmptyMail(MailSender.java:389)
at hudson.tasks.MailSender.createFailureMail(MailSender.java:266)
at hudson.tasks.MailSender.createMail(MailSender.java:181)
at hudson.tasks.MailSender.run(MailSender.java:112)
at hudson.tasks.Mailer.perform(Mailer.java:170)
at 
hudson.tasks.BuildStepCompatibilityLayer.perform(BuildStepCompatibilityLayer.java:81)
at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20)
at 
hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:736)
at 
hudson.model.AbstractBuild$AbstractBuildExecution.performAllBuildSteps(AbstractBuild.java:682)
at hudson.model.Build$BuildExecution.post2(Build.java:186)
at 
hudson.model.AbstractBuild$AbstractBuildExecution.post(AbstractBuild.java:627)
at hudson.model.Run.execute(Run.java:1762)
at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
at hudson.model.ResourceController.execute(ResourceController.java:97)
at hudson.model.Executor.run(Executor.java:419)

Build failed in Jenkins: beam_PostCommit_Python_ValidatesRunner_Dataflow #330

[Apache Jenkins Server]

See 


--
Started by timer
[EnvInject] - Loading node environment variables.
Building remotely on beam3 (beam) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* 
 > +refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/*
 > git rev-parse origin/master^{commit} # timeout=10
Checking out Revision d6345e52e6e8b319456cbad6ce11b05399815930 (origin/master)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f d6345e52e6e8b319456cbad6ce11b05399815930
Commit message: "Update scm to use gitbox.apache.org"
 > git rev-list d6345e52e6e8b319456cbad6ce11b05399815930 # timeout=10
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
[EnvInject] - Executing scripts and injecting environment variables after the 
SCM step.
[EnvInject] - Injecting as environment variables the properties content 
SPARK_LOCAL_IP=127.0.0.1

[EnvInject] - Variables injected successfully.
[beam_PostCommit_Python_ValidatesRunner_Dataflow] $ /bin/bash -xe 
/tmp/jenkins8601408996096768149.sh
+ bash sdks/python/run_validatesrunner.sh
bash: sdks/python/run_validatesrunner.sh: No such file or directory
Build step 'Execute shell' marked build as failure

Build failed in Jenkins: beam_PerformanceTests_Spark #1044

[Apache Jenkins Server]

See 


--
Started by timer
[EnvInject] - Loading node environment variables.
Building remotely on beam2 (beam) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* 
 > +refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/*
 > git rev-parse origin/master^{commit} # timeout=10
Checking out Revision d6345e52e6e8b319456cbad6ce11b05399815930 (origin/master)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f d6345e52e6e8b319456cbad6ce11b05399815930
Commit message: "Update scm to use gitbox.apache.org"
 > git rev-list d6345e52e6e8b319456cbad6ce11b05399815930 # timeout=10
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
[EnvInject] - Executing scripts and injecting environment variables after the 
SCM step.
[EnvInject] - Injecting as environment variables the properties content 
SPARK_LOCAL_IP=127.0.0.1

[EnvInject] - Variables injected successfully.
[beam_PerformanceTests_Spark] $ /bin/bash -xe /tmp/jenkins7190638623142391359.sh
+ rm -rf PerfKitBenchmarker
[beam_PerformanceTests_Spark] $ /bin/bash -xe /tmp/jenkins7672099054652873137.sh
+ git clone https://github.com/GoogleCloudPlatform/PerfKitBenchmarker.git
Cloning into 'PerfKitBenchmarker'...
[beam_PerformanceTests_Spark] $ /bin/bash -xe /tmp/jenkins4137124378413819013.sh
+ pip install --user -r PerfKitBenchmarker/requirements.txt
Requirement already satisfied: absl-py in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 14))
Requirement already satisfied: jinja2>=2.7 in 
/usr/local/lib/python2.7/dist-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 15))
Requirement already satisfied: setuptools in /usr/lib/python2.7/dist-packages 
(from -r PerfKitBenchmarker/requirements.txt (line 16))
Requirement already satisfied: colorlog[windows]==2.6.0 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 17))
Requirement already satisfied: blinker>=1.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 18))
Requirement already satisfied: futures>=3.0.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 19))
Requirement already satisfied: PyYAML==3.12 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 20))
Requirement already satisfied: pint>=0.7 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 21))
Requirement already satisfied: numpy in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 22))
Requirement already satisfied: functools32 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 23))
Requirement already satisfied: contextlib2>=0.5.1 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 24))
Requirement already satisfied: pywinrm in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 25))
Requirement already satisfied: six in 
/home/jenkins/.local/lib/python2.7/site-packages (from absl-py->-r 
PerfKitBenchmarker/requirements.txt (line 14))
Requirement already satisfied: MarkupSafe in 
/usr/local/lib/python2.7/dist-packages (from jinja2>=2.7->-r 
PerfKitBenchmarker/requirements.txt (line 15))
Requirement already satisfied: colorama; extra == "windows" in 
/usr/lib/python2.7/dist-packages (from colorlog[windows]==2.6.0->-r 
PerfKitBenchmarker/requirements.txt (line 17))
Requirement already satisfied: xmltodict in 
/home/jenkins/.local/lib/python2.7/site-packages (from pywinrm->-r 
PerfKitBenchmarker/requirements.txt (line 25))
Requirement already satisfied: requests-ntlm>=0.3.0 in 
/home/jenkins/.local/lib/python2.7/site-packages (from pywinrm->-r 
PerfKitBenchmarker/requirements.txt (line 25))
Requirement already satisfied: requests>=2.9.1 in 
/home/jenkins/.local/lib/python2.7/site-packages (from pywinrm->-r 
PerfKitBenchmarker/requirements.txt (line 25))
Requirement already satisfied: ntlm-auth>=1.0.2 in 
/home/jenkins/.local/lib/python2.7/site-packages (from 
requests-ntlm>=0.3.0->pywinrm->-r PerfKitBenchmarker/requirements.txt (line 25))
Requirement already satisfied: cryptography>=1.3 in 

Build failed in Jenkins: beam_PerformanceTests_Python #597

[Apache Jenkins Server]

See 


--
Started by timer
[EnvInject] - Loading node environment variables.
Building remotely on beam7 (beam) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* 
 > +refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/*
 > git rev-parse origin/master^{commit} # timeout=10
Checking out Revision d6345e52e6e8b319456cbad6ce11b05399815930 (origin/master)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f d6345e52e6e8b319456cbad6ce11b05399815930
Commit message: "Update scm to use gitbox.apache.org"
 > git rev-list d6345e52e6e8b319456cbad6ce11b05399815930 # timeout=10
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
[EnvInject] - Executing scripts and injecting environment variables after the 
SCM step.
[EnvInject] - Injecting as environment variables the properties content 
SPARK_LOCAL_IP=127.0.0.1

[EnvInject] - Variables injected successfully.
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins9050465276054033552.sh
+ rm -rf PerfKitBenchmarker
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins1290482260441972051.sh
+ git clone https://github.com/GoogleCloudPlatform/PerfKitBenchmarker.git
Cloning into 'PerfKitBenchmarker'...
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins9185759867182903973.sh
+ pip install --user -r PerfKitBenchmarker/requirements.txt
Requirement already satisfied: absl-py in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 14))
Requirement already satisfied: jinja2>=2.7 in 
/usr/local/lib/python2.7/dist-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 15))
Requirement already satisfied: setuptools in /usr/lib/python2.7/dist-packages 
(from -r PerfKitBenchmarker/requirements.txt (line 16))
Requirement already satisfied: colorlog[windows]==2.6.0 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 17))
Requirement already satisfied: blinker>=1.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 18))
Requirement already satisfied: futures>=3.0.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 19))
Requirement already satisfied: PyYAML==3.12 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 20))
Requirement already satisfied: pint>=0.7 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 21))
Requirement already satisfied: numpy in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 22))
Requirement already satisfied: functools32 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 23))
Requirement already satisfied: contextlib2>=0.5.1 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 24))
Requirement already satisfied: pywinrm in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 25))
Requirement already satisfied: six in 
/home/jenkins/.local/lib/python2.7/site-packages (from absl-py->-r 
PerfKitBenchmarker/requirements.txt (line 14))
Requirement already satisfied: MarkupSafe>=0.23 in 
/usr/local/lib/python2.7/dist-packages (from jinja2>=2.7->-r 
PerfKitBenchmarker/requirements.txt (line 15))
Requirement already satisfied: colorama; extra == "windows" in 
/usr/lib/python2.7/dist-packages (from colorlog[windows]==2.6.0->-r 
PerfKitBenchmarker/requirements.txt (line 17))
Requirement already satisfied: xmltodict in 
/home/jenkins/.local/lib/python2.7/site-packages (from pywinrm->-r 
PerfKitBenchmarker/requirements.txt (line 25))
Requirement already satisfied: requests-ntlm>=0.3.0 in 
/home/jenkins/.local/lib/python2.7/site-packages (from pywinrm->-r 
PerfKitBenchmarker/requirements.txt (line 25))
Requirement already satisfied: requests>=2.9.1 in 
/home/jenkins/.local/lib/python2.7/site-packages (from pywinrm->-r 
PerfKitBenchmarker/requirements.txt (line 25))
Requirement already satisfied: ntlm-auth>=1.0.2 in 
/home/jenkins/.local/lib/python2.7/site-packages (from 
requests-ntlm>=0.3.0->pywinrm->-r PerfKitBenchmarker/requirements.txt (line 25))
Requirement already satisfied: 

[jira] [Created] (BEAM-3247) Sample.any memory constraint

[Neville Li (JIRA)]

Neville Li created BEAM-3247:


 Summary: Sample.any memory constraint
 Key: BEAM-3247
 URL: https://issues.apache.org/jira/browse/BEAM-3247
 Project: Beam
  Issue Type: Improvement
  Components: sdk-java-core
Affects Versions: 2.1.0
Reporter: Neville Li
Assignee: Kenneth Knowles
Priority: Minor


Right now {{Sample.any}} converts the collection to an iterable view and take 
first n in a side input. This may require materializing the entire collection 
to disk and is potentially inefficient.
https://github.com/apache/beam/blob/v2.1.0/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/Sample.java#L74

It can be fixed by applying a truncating `DoFn` first, then a combine into 
`List` which limits the list size, and finally flattening the list.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Assigned] (BEAM-3247) Sample.any memory constraint

[Neville Li (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3247?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Neville Li reassigned BEAM-3247:


Assignee: Neville Li  (was: Kenneth Knowles)

> Sample.any memory constraint
> 
>
> Key: BEAM-3247
> URL: https://issues.apache.org/jira/browse/BEAM-3247
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-core
>Affects Versions: 2.1.0
>Reporter: Neville Li
>Assignee: Neville Li
>Priority: Minor
>
> Right now {{Sample.any}} converts the collection to an iterable view and take 
> first n in a side input. This may require materializing the entire collection 
> to disk and is potentially inefficient.
> https://github.com/apache/beam/blob/v2.1.0/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/Sample.java#L74
> It can be fixed by applying a truncating `DoFn` first, then a combine into 
> `List` which limits the list size, and finally flattening the list.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Build failed in Jenkins: beam_PostCommit_Python_Verify #3608

[Apache Jenkins Server]

See 


--
Started by timer
[EnvInject] - Loading node environment variables.
Building remotely on beam3 (beam) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* 
 > +refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/*
 > git rev-parse origin/master^{commit} # timeout=10
Checking out Revision d6345e52e6e8b319456cbad6ce11b05399815930 (origin/master)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f d6345e52e6e8b319456cbad6ce11b05399815930
Commit message: "Update scm to use gitbox.apache.org"
 > git rev-list d6345e52e6e8b319456cbad6ce11b05399815930 # timeout=10
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
[EnvInject] - Executing scripts and injecting environment variables after the 
SCM step.
[EnvInject] - Injecting as environment variables the properties content 
SPARK_LOCAL_IP=127.0.0.1

[EnvInject] - Variables injected successfully.
[beam_PostCommit_Python_Verify] $ /bin/bash -xe 
/tmp/jenkins2636582913495541688.sh
+ bash sdks/python/run_postcommit.sh
bash: sdks/python/run_postcommit.sh: No such file or directory
Build step 'Execute shell' marked build as failure
Not sending mail to unregistered user ajam...@ajamato2016.sea.corp.google.com
Not sending mail to unregistered user c...@google.com
Not sending mail to unregistered user hero...@google.com
Not sending mail to unregistered user git...@nakomis.com
Not sending mail to unregistered user valen...@google.com
Not sending mail to unregistered user 
ke...@kedin-macbookpro.roam.corp.google.com
Not sending mail to unregistered user xumingmi...@gmail.com
Not sending mail to unregistered user k...@google.com
Not sending mail to unregistered user kirpic...@google.com
Not sending mail to unregistered user ekirpic...@gmail.com
ERROR: Unable to send to address: echauc...@gmail.com+jbono...@apache.org
javax.mail.internet.AddressException: Domain contains illegal character in 
string ``echauc...@gmail.com+jbono...@apache.org''
at 
javax.mail.internet.InternetAddress.checkAddress(InternetAddress.java:1269)
at javax.mail.internet.InternetAddress.parse(InternetAddress.java:1091)
at javax.mail.internet.InternetAddress.parse(InternetAddress.java:633)
at javax.mail.internet.InternetAddress.(InternetAddress.java:111)
at hudson.tasks.Mailer.stringToAddress(Mailer.java:206)
at 
jenkins.plugins.mailer.tasks.MimeMessageBuilder.toNormalizedAddress(MimeMessageBuilder.java:309)
at 
jenkins.plugins.mailer.tasks.MimeMessageBuilder.addRecipients(MimeMessageBuilder.java:168)
at 
jenkins.plugins.mailer.tasks.MimeMessageBuilder.addRecipients(MimeMessageBuilder.java:160)
at hudson.tasks.MailSender.createEmptyMail(MailSender.java:389)
at hudson.tasks.MailSender.createFailureMail(MailSender.java:266)
at hudson.tasks.MailSender.createMail(MailSender.java:181)
at hudson.tasks.MailSender.run(MailSender.java:112)
at hudson.tasks.Mailer.perform(Mailer.java:170)
at 
hudson.tasks.BuildStepCompatibilityLayer.perform(BuildStepCompatibilityLayer.java:81)
at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20)
at 
hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:736)
at 
hudson.model.AbstractBuild$AbstractBuildExecution.performAllBuildSteps(AbstractBuild.java:682)
at hudson.model.Build$BuildExecution.post2(Build.java:186)
at 
hudson.model.AbstractBuild$AbstractBuildExecution.post(AbstractBuild.java:627)
at hudson.model.Run.execute(Run.java:1762)
at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
at hudson.model.ResourceController.execute(ResourceController.java:97)
at hudson.model.Executor.run(Executor.java:419)

Build failed in Jenkins: beam_PostCommit_Python_ValidatesRunner_Dataflow #329

[Apache Jenkins Server]

See 


--
Started by timer
[EnvInject] - Loading node environment variables.
Building remotely on beam3 (beam) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* 
 > +refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/*
 > git rev-parse origin/master^{commit} # timeout=10
Checking out Revision d6345e52e6e8b319456cbad6ce11b05399815930 (origin/master)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f d6345e52e6e8b319456cbad6ce11b05399815930
Commit message: "Update scm to use gitbox.apache.org"
 > git rev-list d6345e52e6e8b319456cbad6ce11b05399815930 # timeout=10
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
[EnvInject] - Executing scripts and injecting environment variables after the 
SCM step.
[EnvInject] - Injecting as environment variables the properties content 
SPARK_LOCAL_IP=127.0.0.1

[EnvInject] - Variables injected successfully.
[beam_PostCommit_Python_ValidatesRunner_Dataflow] $ /bin/bash -xe 
/tmp/jenkins5934036575706814946.sh
+ bash sdks/python/run_validatesrunner.sh
bash: sdks/python/run_validatesrunner.sh: No such file or directory
Build step 'Execute shell' marked build as failure

Build failed in Jenkins: beam_PerformanceTests_Spark #1043

[Apache Jenkins Server]

See 


--
Started by timer
[EnvInject] - Loading node environment variables.
Building remotely on beam4 (beam) in workspace 

Cloning the remote Git repository
Cloning repository https://github.com/apache/beam.git
 > git init  
 > # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/*
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
 > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # 
 > timeout=10
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* 
 > +refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/*
 > git rev-parse origin/master^{commit} # timeout=10
Checking out Revision d6345e52e6e8b319456cbad6ce11b05399815930 (origin/master)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f d6345e52e6e8b319456cbad6ce11b05399815930
Commit message: "Update scm to use gitbox.apache.org"
 > git rev-list d6345e52e6e8b319456cbad6ce11b05399815930 # timeout=10
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
[EnvInject] - Executing scripts and injecting environment variables after the 
SCM step.
[EnvInject] - Injecting as environment variables the properties content 
SPARK_LOCAL_IP=127.0.0.1

[EnvInject] - Variables injected successfully.
[beam_PerformanceTests_Spark] $ /bin/bash -xe /tmp/jenkins5746185760577648949.sh
+ rm -rf PerfKitBenchmarker
[beam_PerformanceTests_Spark] $ /bin/bash -xe /tmp/jenkins528365363995897363.sh
+ git clone https://github.com/GoogleCloudPlatform/PerfKitBenchmarker.git
Cloning into 'PerfKitBenchmarker'...
[beam_PerformanceTests_Spark] $ /bin/bash -xe /tmp/jenkins2498235961703173733.sh
+ pip install --user -r PerfKitBenchmarker/requirements.txt
Requirement already satisfied: absl-py in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 14))
Requirement already satisfied: jinja2>=2.7 in 
/usr/local/lib/python2.7/dist-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 15))
Requirement already satisfied: setuptools in /usr/lib/python2.7/dist-packages 
(from -r PerfKitBenchmarker/requirements.txt (line 16))
Requirement already satisfied: colorlog[windows]==2.6.0 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 17))
Requirement already satisfied: blinker>=1.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 18))
Requirement already satisfied: futures>=3.0.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 19))
Requirement already satisfied: PyYAML==3.12 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 20))
Requirement already satisfied: pint>=0.7 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 21))
Requirement already satisfied: numpy in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 22))
Requirement already satisfied: functools32 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 23))
Requirement already satisfied: contextlib2>=0.5.1 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 24))
Requirement already satisfied: pywinrm in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 25))
Requirement already satisfied: six in 
/home/jenkins/.local/lib/python2.7/site-packages (from absl-py->-r 
PerfKitBenchmarker/requirements.txt (line 14))
Requirement already satisfied: MarkupSafe>=0.23 in 
/usr/local/lib/python2.7/dist-packages (from jinja2>=2.7->-r 
PerfKitBenchmarker/requirements.txt (line 15))
Requirement already satisfied: colorama; extra == "windows" in 
/usr/lib/python2.7/dist-packages (from colorlog[windows]==2.6.0->-r 
PerfKitBenchmarker/requirements.txt (line 17))
Requirement already satisfied: xmltodict in 
/home/jenkins/.local/lib/python2.7/site-packages (from pywinrm->-r 
PerfKitBenchmarker/requirements.txt (line 25))
Requirement already satisfied: requests-ntlm>=0.3.0 in 
/home/jenkins/.local/lib/python2.7/site-packages (from pywinrm->-r 

Build failed in Jenkins: beam_PerformanceTests_Python #596

[Apache Jenkins Server]

See 


--
Started by timer
[EnvInject] - Loading node environment variables.
Building remotely on beam7 (beam) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* 
 > +refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/*
 > git rev-parse origin/master^{commit} # timeout=10
Checking out Revision d6345e52e6e8b319456cbad6ce11b05399815930 (origin/master)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f d6345e52e6e8b319456cbad6ce11b05399815930
Commit message: "Update scm to use gitbox.apache.org"
 > git rev-list d6345e52e6e8b319456cbad6ce11b05399815930 # timeout=10
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
[EnvInject] - Executing scripts and injecting environment variables after the 
SCM step.
[EnvInject] - Injecting as environment variables the properties content 
SPARK_LOCAL_IP=127.0.0.1

[EnvInject] - Variables injected successfully.
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins3374127833000810980.sh
+ rm -rf PerfKitBenchmarker
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins667310352700317.sh
+ git clone https://github.com/GoogleCloudPlatform/PerfKitBenchmarker.git
Cloning into 'PerfKitBenchmarker'...
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins5192022671210824452.sh
+ pip install --user -r PerfKitBenchmarker/requirements.txt
Requirement already satisfied: absl-py in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 14))
Requirement already satisfied: jinja2>=2.7 in 
/usr/local/lib/python2.7/dist-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 15))
Requirement already satisfied: setuptools in /usr/lib/python2.7/dist-packages 
(from -r PerfKitBenchmarker/requirements.txt (line 16))
Requirement already satisfied: colorlog[windows]==2.6.0 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 17))
Requirement already satisfied: blinker>=1.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 18))
Requirement already satisfied: futures>=3.0.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 19))
Requirement already satisfied: PyYAML==3.12 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 20))
Requirement already satisfied: pint>=0.7 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 21))
Requirement already satisfied: numpy in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 22))
Requirement already satisfied: functools32 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 23))
Requirement already satisfied: contextlib2>=0.5.1 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 24))
Requirement already satisfied: pywinrm in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 25))
Requirement already satisfied: six in 
/home/jenkins/.local/lib/python2.7/site-packages (from absl-py->-r 
PerfKitBenchmarker/requirements.txt (line 14))
Requirement already satisfied: MarkupSafe>=0.23 in 
/usr/local/lib/python2.7/dist-packages (from jinja2>=2.7->-r 
PerfKitBenchmarker/requirements.txt (line 15))
Requirement already satisfied: colorama; extra == "windows" in 
/usr/lib/python2.7/dist-packages (from colorlog[windows]==2.6.0->-r 
PerfKitBenchmarker/requirements.txt (line 17))
Requirement already satisfied: xmltodict in 
/home/jenkins/.local/lib/python2.7/site-packages (from pywinrm->-r 
PerfKitBenchmarker/requirements.txt (line 25))
Requirement already satisfied: requests-ntlm>=0.3.0 in 
/home/jenkins/.local/lib/python2.7/site-packages (from pywinrm->-r 
PerfKitBenchmarker/requirements.txt (line 25))
Requirement already satisfied: requests>=2.9.1 in 
/home/jenkins/.local/lib/python2.7/site-packages (from pywinrm->-r 
PerfKitBenchmarker/requirements.txt (line 25))
Requirement already satisfied: ntlm-auth>=1.0.2 in 
/home/jenkins/.local/lib/python2.7/site-packages (from 
requests-ntlm>=0.3.0->pywinrm->-r PerfKitBenchmarker/requirements.txt (line 25))
Requirement already satisfied: 

[jira] [Created] (BEAM-3246) BigtableIO should merge splits if they exceed 15K

[Solomon Duskis (JIRA)]

Solomon Duskis created BEAM-3246:


 Summary: BigtableIO should merge splits if they exceed 15K
 Key: BEAM-3246
 URL: https://issues.apache.org/jira/browse/BEAM-3246
 Project: Beam
  Issue Type: Bug
  Components: sdk-java-gcp
Reporter: Solomon Duskis
Assignee: Solomon Duskis


A customer hit a problem with a large number of splits.  CloudBitableIO fixes 
that here 
https://github.com/GoogleCloudPlatform/cloud-bigtable-client/blob/master/bigtable-dataflow-parent/bigtable-hbase-beam/src/main/java/com/google/cloud/bigtable/beam/CloudBigtableIO.java#L241

BigtableIO should have similar logic.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (BEAM-3244) Flink runner does not respect ParDo's lifecycle on case of exceptions

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-3244?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16265467#comment-16265467
 ] 

ASF GitHub Bot commented on BEAM-3244:
--

iemejia commented on a change in pull request #4174: [BEAM-3244] Ensure 
execution of teardown method on Flink's DoFnOperator
URL: https://github.com/apache/beam/pull/4174#discussion_r153012280
 
 

 ##
 File path: 
runners/flink/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/DoFnOperator.java
 ##
 @@ -380,7 +386,6 @@ public void close() throws Exception {
   }
 }
 checkFinishBundleTimer.cancel(true);
 
 Review comment:
   @aljoscha I had the doubt if this one should be moved to dispose too, given 
that close can eventually not be called. WDYT ?


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Flink runner does not respect ParDo's lifecycle on case of exceptions
> -
>
> Key: BEAM-3244
> URL: https://issues.apache.org/jira/browse/BEAM-3244
> Project: Beam
>  Issue Type: Bug
>  Components: runner-flink
>Affects Versions: 2.2.0
>Reporter: Ismaël Mejía
>Assignee: Ismaël Mejía
>
> The lifecycle of the DoFn is not respected in case of exception in any of the 
> lifecycle methods after setup.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (BEAM-3244) Flink runner does not respect ParDo's lifecycle on case of exceptions

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-3244?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16265465#comment-16265465
 ] 

ASF GitHub Bot commented on BEAM-3244:
--

iemejia commented on issue #4174: [BEAM-3244] Ensure execution of teardown 
method on Flink's DoFnOperator
URL: https://github.com/apache/beam/pull/4174#issuecomment-346875689
 
 
   Run Flink ValidatesRunner


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Flink runner does not respect ParDo's lifecycle on case of exceptions
> -
>
> Key: BEAM-3244
> URL: https://issues.apache.org/jira/browse/BEAM-3244
> Project: Beam
>  Issue Type: Bug
>  Components: runner-flink
>Affects Versions: 2.2.0
>Reporter: Ismaël Mejía
>Assignee: Ismaël Mejía
>
> The lifecycle of the DoFn is not respected in case of exception in any of the 
> lifecycle methods after setup.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (BEAM-3244) Flink runner does not respect ParDo's lifecycle on case of exceptions

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-3244?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16265463#comment-16265463
 ] 

ASF GitHub Bot commented on BEAM-3244:
--

GitHub user iemejia opened a pull request:

https://github.com/apache/beam/pull/4174

[BEAM-3244] Ensure execution of teardown method on Flink's DoFnOperator

Follow this checklist to help us incorporate your contribution quickly and 
easily:

 - [x] Make sure there is a [JIRA 
issue](https://issues.apache.org/jira/projects/BEAM/issues/) filed for the 
change (usually before you start working on it).  Trivial changes like typos do 
not require a JIRA issue.  Your pull request should address just this issue, 
without pulling in other changes.
 - [x] Each commit in the pull request should have a meaningful subject 
line and body.
 - [x] Format the pull request title like `[BEAM-XXX] Fixes bug in 
ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA 
issue.
 - [x] Write a pull request description that is detailed enough to 
understand what the pull request does, how, and why.
 - [x] Run `mvn clean verify` to make sure basic checks pass. A more 
thorough check will be performed on your pull request automatically.
 - [ ] If this contribution is large, please file an Apache [Individual 
Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).

---


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/iemejia/beam BEAM-3244-dofn-flink

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/4174.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #4174


commit c72084d3f08e92ddb7adb5199ed3fb757fc344cf
Author: Ismaël Mejía 
Date:   2017-11-24T17:31:51Z

[BEAM-3244] Ensure execution of teardown method on Flink's DoFnOperator




> Flink runner does not respect ParDo's lifecycle on case of exceptions
> -
>
> Key: BEAM-3244
> URL: https://issues.apache.org/jira/browse/BEAM-3244
> Project: Beam
>  Issue Type: Bug
>  Components: runner-flink
>Affects Versions: 2.2.0
>Reporter: Ismaël Mejía
>Assignee: Ismaël Mejía
>
> The lifecycle of the DoFn is not respected in case of exception in any of the 
> lifecycle methods after setup.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (BEAM-3244) Flink runner does not respect ParDo's lifecycle on case of exceptions

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-3244?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16265462#comment-16265462
 ] 

ASF GitHub Bot commented on BEAM-3244:
--

iemejia opened a new pull request #4174: [BEAM-3244] Ensure execution of 
teardown method on Flink's DoFnOperator
URL: https://github.com/apache/beam/pull/4174
 
 
   Follow this checklist to help us incorporate your contribution quickly and 
easily:
   
- [x] Make sure there is a [JIRA 
issue](https://issues.apache.org/jira/projects/BEAM/issues/) filed for the 
change (usually before you start working on it).  Trivial changes like typos do 
not require a JIRA issue.  Your pull request should address just this issue, 
without pulling in other changes.
- [x] Each commit in the pull request should have a meaningful subject line 
and body.
- [x] Format the pull request title like `[BEAM-XXX] Fixes bug in 
ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA 
issue.
- [x] Write a pull request description that is detailed enough to 
understand what the pull request does, how, and why.
- [x] Run `mvn clean verify` to make sure basic checks pass. A more 
thorough check will be performed on your pull request automatically.
- [ ] If this contribution is large, please file an Apache [Individual 
Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
   
   ---
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Flink runner does not respect ParDo's lifecycle on case of exceptions
> -
>
> Key: BEAM-3244
> URL: https://issues.apache.org/jira/browse/BEAM-3244
> Project: Beam
>  Issue Type: Bug
>  Components: runner-flink
>Affects Versions: 2.2.0
>Reporter: Ismaël Mejía
>Assignee: Ismaël Mejía
>
> The lifecycle of the DoFn is not respected in case of exception in any of the 
> lifecycle methods after setup.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (BEAM-3242) Gearpump runner does not respect ParDo's lifecycle on case of exceptions

[JIRA]

 [ 
https://issues.apache.org/jira/browse/BEAM-3242?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismaël Mejía updated BEAM-3242:
---
Affects Version/s: (was: 2.1.0)

> Gearpump runner does not respect ParDo's lifecycle on case of exceptions
> 
>
> Key: BEAM-3242
> URL: https://issues.apache.org/jira/browse/BEAM-3242
> Project: Beam
>  Issue Type: Bug
>  Components: runner-gearpump
>Affects Versions: 2.2.0
>Reporter: Ismaël Mejía
>Assignee: Manu Zhang
>Priority: Minor
>
> The lifecycle of the DoFn is not respected in case of exception in any of the 
> lifecycle methods after setup.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (BEAM-3244) Flink runner does not respect ParDo's lifecycle on case of exceptions

[JIRA]

 [ 
https://issues.apache.org/jira/browse/BEAM-3244?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismaël Mejía updated BEAM-3244:
---
Affects Version/s: (was: 2.1.0)
   2.2.0

> Flink runner does not respect ParDo's lifecycle on case of exceptions
> -
>
> Key: BEAM-3244
> URL: https://issues.apache.org/jira/browse/BEAM-3244
> Project: Beam
>  Issue Type: Bug
>  Components: runner-flink
>Affects Versions: 2.2.0
>Reporter: Ismaël Mejía
>Assignee: Ismaël Mejía
>
> The lifecycle of the DoFn is not respected in case of exception in any of the 
> lifecycle methods after setup.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (BEAM-3187) Spark runner does not respect ParDo's lifecycle on case of exceptions

[JIRA]

 [ 
https://issues.apache.org/jira/browse/BEAM-3187?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismaël Mejía updated BEAM-3187:
---
Affects Version/s: (was: 2.1.0)

> Spark runner does not respect ParDo's lifecycle on case of exceptions
> -
>
> Key: BEAM-3187
> URL: https://issues.apache.org/jira/browse/BEAM-3187
> Project: Beam
>  Issue Type: Bug
>  Components: runner-spark
>Affects Versions: 2.2.0
>Reporter: Ismaël Mejía
>Assignee: Ismaël Mejía
>
> The lifecycle of the DoFn is not respected in the case of an Exception on the 
> finishBundle method. The Spark runner does not call the teardown function of 
> the DoFn as it should since this should be dual with the setup.
> This happens too if the exception happens in the startBundle too.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (BEAM-3241) Apex runner does not respect ParDo's lifecycle on case of exceptions

[JIRA]

 [ 
https://issues.apache.org/jira/browse/BEAM-3241?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismaël Mejía updated BEAM-3241:
---
Affects Version/s: (was: 2.1.0)

> Apex runner does not respect ParDo's lifecycle on case of exceptions
> 
>
> Key: BEAM-3241
> URL: https://issues.apache.org/jira/browse/BEAM-3241
> Project: Beam
>  Issue Type: Bug
>  Components: runner-apex
>Affects Versions: 2.2.0
>Reporter: Ismaël Mejía
>Assignee: Thomas Weise
>Priority: Minor
>
> The lifecycle of the DoFn is not respected in case of exception in any of the 
> lifecycle methods after setup.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (BEAM-3245) Dataflow runner does not respect ParDo's lifecycle on case of exceptions

[JIRA]

 [ 
https://issues.apache.org/jira/browse/BEAM-3245?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismaël Mejía updated BEAM-3245:
---
Affects Version/s: (was: 2.1.0)

> Dataflow runner does not respect ParDo's lifecycle on case of exceptions
> 
>
> Key: BEAM-3245
> URL: https://issues.apache.org/jira/browse/BEAM-3245
> Project: Beam
>  Issue Type: Bug
>  Components: runner-dataflow
>Affects Versions: 2.2.0
>Reporter: Ismaël Mejía
>Assignee: Thomas Groh
>
> The lifecycle of the DoFn is not respected in case of exception in any of the 
> lifecycle methods after setup.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (BEAM-3187) Spark runner does not respect ParDo's lifecycle on case of exceptions

[JIRA]

 [ 
https://issues.apache.org/jira/browse/BEAM-3187?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismaël Mejía updated BEAM-3187:
---
Affects Version/s: 2.2.0

> Spark runner does not respect ParDo's lifecycle on case of exceptions
> -
>
> Key: BEAM-3187
> URL: https://issues.apache.org/jira/browse/BEAM-3187
> Project: Beam
>  Issue Type: Bug
>  Components: runner-spark
>Affects Versions: 2.1.0, 2.2.0
>Reporter: Ismaël Mejía
>Assignee: Ismaël Mejía
>
> The lifecycle of the DoFn is not respected in the case of an Exception on the 
> finishBundle method. The Spark runner does not call the teardown function of 
> the DoFn as it should since this should be dual with the setup.
> This happens too if the exception happens in the startBundle too.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (BEAM-3241) Apex runner does not respect ParDo's lifecycle on case of exceptions

[JIRA]

 [ 
https://issues.apache.org/jira/browse/BEAM-3241?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismaël Mejía updated BEAM-3241:
---
Affects Version/s: 2.2.0

> Apex runner does not respect ParDo's lifecycle on case of exceptions
> 
>
> Key: BEAM-3241
> URL: https://issues.apache.org/jira/browse/BEAM-3241
> Project: Beam
>  Issue Type: Bug
>  Components: runner-apex
>Affects Versions: 2.1.0, 2.2.0
>Reporter: Ismaël Mejía
>Assignee: Thomas Weise
>Priority: Minor
>
> The lifecycle of the DoFn is not respected in case of exception in any of the 
> lifecycle methods after setup.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (BEAM-3245) Dataflow runner does not respect ParDo's lifecycle on case of exceptions

[JIRA]

Ismaël Mejía created BEAM-3245:
--

 Summary: Dataflow runner does not respect ParDo's lifecycle on 
case of exceptions
 Key: BEAM-3245
 URL: https://issues.apache.org/jira/browse/BEAM-3245
 Project: Beam
  Issue Type: Bug
  Components: runner-dataflow
Affects Versions: 2.1.0, 2.2.0
Reporter: Ismaël Mejía
Assignee: Thomas Groh


The lifecycle of the DoFn is not respected in case of exception in any of the 
lifecycle methods after setup.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (BEAM-3242) Gearpump runner does not respect ParDo's lifecycle on case of exceptions

[JIRA]

 [ 
https://issues.apache.org/jira/browse/BEAM-3242?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismaël Mejía updated BEAM-3242:
---
Affects Version/s: 2.2.0

> Gearpump runner does not respect ParDo's lifecycle on case of exceptions
> 
>
> Key: BEAM-3242
> URL: https://issues.apache.org/jira/browse/BEAM-3242
> Project: Beam
>  Issue Type: Bug
>  Components: runner-gearpump
>Affects Versions: 2.1.0, 2.2.0
>Reporter: Ismaël Mejía
>Assignee: Manu Zhang
>Priority: Minor
>
> The lifecycle of the DoFn is not respected in case of exception in any of the 
> lifecycle methods after setup.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (BEAM-3244) Flink runner does not respect ParDo's lifecycle on case of exceptions

[JIRA]

Ismaël Mejía created BEAM-3244:
--

 Summary: Flink runner does not respect ParDo's lifecycle on case 
of exceptions
 Key: BEAM-3244
 URL: https://issues.apache.org/jira/browse/BEAM-3244
 Project: Beam
  Issue Type: Bug
  Components: runner-flink
Affects Versions: 2.1.0
Reporter: Ismaël Mejía
Assignee: Ismaël Mejía


The lifecycle of the DoFn is not respected in case of exception in any of the 
lifecycle methods after setup.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (BEAM-3060) Add performance tests for commonly used file-based I/O PTransforms

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-3060?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16265420#comment-16265420
 ] 

ASF GitHub Bot commented on BEAM-3060:
--

lgajowy commented on issue #4149: [BEAM-3060] Add Compressed TextIOIT
URL: https://github.com/apache/beam/pull/4149#issuecomment-346866400
 
 
   @chamikaramj thanks for the review! Here's another batch of changes, as 
commented above. 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Add performance tests for commonly used file-based I/O PTransforms
> --
>
> Key: BEAM-3060
> URL: https://issues.apache.org/jira/browse/BEAM-3060
> Project: Beam
>  Issue Type: Test
>  Components: sdk-java-core
>Reporter: Chamikara Jayalath
>Assignee: Szymon Nieradka
>
> We recently added a performance testing framework [1] that can be used to do 
> following.
> (1) Execute Beam tests using PerfkitBenchmarker
> (2) Manage Kubernetes-based deployments of data stores.
> (3) Easily publish benchmark results. 
> I think it will be useful to add performance tests for commonly used 
> file-based I/O PTransforms using this framework. I suggest looking into 
> following formats initially.
> (1) AvroIO
> (2) TextIO
> (3) Compressed text using TextIO
> (4) TFRecordIO
> It should be possibly to run these tests for various Beam runners (Direct, 
> Dataflow, Flink, Spark, etc.) and file-systems (GCS, local, HDFS, etc.) 
> easily.
> In the initial version, tests can be made manually triggerable for PRs 
> through Jenkins. Later, we could make some of these tests run periodically 
> and publish benchmark results (to BigQuery) through PerfkitBenchmarker.
> [1] https://beam.apache.org/documentation/io/testing/



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (BEAM-3060) Add performance tests for commonly used file-based I/O PTransforms

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-3060?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16265393#comment-16265393
 ] 

ASF GitHub Bot commented on BEAM-3060:
--

lgajowy commented on a change in pull request #4149: [BEAM-3060] Add Compressed 
TextIOIT
URL: https://github.com/apache/beam/pull/4149#discussion_r152999363
 
 

 ##
 File path: 
sdks/java/io/file-based-io-tests/src/test/java/org/apache/beam/sdk/io/text/TextIOIT.java
 ##
 @@ -83,25 +90,82 @@ private static String appendTimestamp(String 
filenamePrefix) {
 return String.format("%s_%s", filenamePrefix, new Date().getTime());
   }
 
-  @Test
-  public void writeThenReadAll() {
-PCollection testFilenames = pipeline
-.apply("Generate sequence", 
GenerateSequence.from(0).to(numberOfTextLines))
-.apply("Produce text lines", ParDo.of(new 
DeterministicallyConstructTestTextLineFn()))
-.apply("Write content to files", 
TextIO.write().to(filenamePrefix).withOutputFilenames())
-.getPerDestinationOutputFilenames().apply(Values.create());
+  /** IO IT with no compression. */
+  @RunWith(JUnit4.class)
+  public static class UncompressedTextIOIT {
+
+@Rule
+public TestPipeline pipeline = TestPipeline.create();
+
+@Test
+public void writeThenReadAll() {
+  PCollection testFilenames = pipeline
+  .apply("Generate sequence", 
GenerateSequence.from(0).to(numberOfTextLines))
+  .apply("Produce text lines", ParDo.of(new 
DeterministicallyConstructTestTextLineFn()))
+  .apply("Write content to files", 
TextIO.write().to(filenamePrefix).withOutputFilenames())
+  .getPerDestinationOutputFilenames().apply(Values.create());
+
+  PCollection consolidatedHashcode = testFilenames
+  .apply("Read all files", TextIO.readAll())
+  .apply("Calculate hashcode", Combine.globally(new HashingFn()));
+
+  String expectedHash = getExpectedHashForLineCount(numberOfTextLines);
+  PAssert.thatSingleton(consolidatedHashcode).isEqualTo(expectedHash);
+
+  testFilenames.apply("Delete test files", ParDo.of(new DeleteFileFn())
+  
.withSideInputs(consolidatedHashcode.apply(View.asSingleton(;
+
+  pipeline.run().waitUntilFinish();
+}
+  }
+
+  /** IO IT with various compression types. */
+  @RunWith(Parameterized.class)
+  public static class CompressedTextIOIT {
+
+@Rule
+public TestPipeline pipeline = TestPipeline.create();
+
+@Parameterized.Parameters()
+public static Iterable data() {
+  return ImmutableList.builder()
+  .add(GZIP)
+  .add(DEFLATE)
+  .add(BZIP2)
+  .build();
+}
+
+@Parameterized.Parameter()
+public Compression compression;
+
+@Test
+public void writeThenReadAllWithCompression() {
+  TextIO.TypedWrite write = TextIO
+  .write()
+  .to(filenamePrefix)
+  .withOutputFilenames()
+  .withCompression(compression);
+
+  TextIO.ReadAll read = TextIO.readAll().withCompression(AUTO);
 
-PCollection consolidatedHashcode = testFilenames
-.apply("Read all files", TextIO.readAll())
-.apply("Calculate hashcode", Combine.globally(new HashingFn()));
+  PCollection testFilenames = pipeline
 
 Review comment:
   I think it's hard to do right now without modifying perfkit's code. As we 
checked, perfkit ignores -D parameters because builds the mvn verify command by 
itself from the parameters passed . I think this could be done in some future 
contribution. We will file a bug report in perfkit soon. 
   
   I think the best solution (at least for now) is to leave the compression 
type in pipeline options. We pass them to perfkit either way (through 
`beam_it_options`) and, what imo is more important, compressionType is very 
test specific (same as numberOfRecords). WDYT?


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Add performance tests for commonly used file-based I/O PTransforms
> --
>
> Key: BEAM-3060
> URL: https://issues.apache.org/jira/browse/BEAM-3060
> Project: Beam
>  Issue Type: Test
>  Components: sdk-java-core
>Reporter: Chamikara Jayalath
>Assignee: Szymon Nieradka
>
> We recently added a performance testing framework [1] that can be used to do 
> following.
> (1) Execute Beam tests using PerfkitBenchmarker
> (2) Manage Kubernetes-based deployments of data stores.
> (3) Easily publish benchmark results. 
> I think it will be useful to add performance tests for commonly used 
> 

[jira] [Commented] (BEAM-3060) Add performance tests for commonly used file-based I/O PTransforms

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-3060?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16265386#comment-16265386
 ] 

ASF GitHub Bot commented on BEAM-3060:
--

lgajowy commented on a change in pull request #4149: [BEAM-3060] Add Compressed 
TextIOIT
URL: https://github.com/apache/beam/pull/4149#discussion_r152998058
 
 

 ##
 File path: 
sdks/java/io/file-based-io-tests/src/test/java/org/apache/beam/sdk/io/text/TextIOIT.java
 ##
 @@ -83,25 +90,82 @@ private static String appendTimestamp(String 
filenamePrefix) {
 return String.format("%s_%s", filenamePrefix, new Date().getTime());
   }
 
-  @Test
-  public void writeThenReadAll() {
-PCollection testFilenames = pipeline
-.apply("Generate sequence", 
GenerateSequence.from(0).to(numberOfTextLines))
-.apply("Produce text lines", ParDo.of(new 
DeterministicallyConstructTestTextLineFn()))
-.apply("Write content to files", 
TextIO.write().to(filenamePrefix).withOutputFilenames())
-.getPerDestinationOutputFilenames().apply(Values.create());
+  /** IO IT with no compression. */
+  @RunWith(JUnit4.class)
+  public static class UncompressedTextIOIT {
 
 Review comment:
   Yes, it works but runs all the 4 tests that are there in the file. But now I 
think this is probably not what we want. This won't be a problem as you 
suggested an even better solution in the comment below. 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Add performance tests for commonly used file-based I/O PTransforms
> --
>
> Key: BEAM-3060
> URL: https://issues.apache.org/jira/browse/BEAM-3060
> Project: Beam
>  Issue Type: Test
>  Components: sdk-java-core
>Reporter: Chamikara Jayalath
>Assignee: Szymon Nieradka
>
> We recently added a performance testing framework [1] that can be used to do 
> following.
> (1) Execute Beam tests using PerfkitBenchmarker
> (2) Manage Kubernetes-based deployments of data stores.
> (3) Easily publish benchmark results. 
> I think it will be useful to add performance tests for commonly used 
> file-based I/O PTransforms using this framework. I suggest looking into 
> following formats initially.
> (1) AvroIO
> (2) TextIO
> (3) Compressed text using TextIO
> (4) TFRecordIO
> It should be possibly to run these tests for various Beam runners (Direct, 
> Dataflow, Flink, Spark, etc.) and file-systems (GCS, local, HDFS, etc.) 
> easily.
> In the initial version, tests can be made manually triggerable for PRs 
> through Jenkins. Later, we could make some of these tests run periodically 
> and publish benchmark results (to BigQuery) through PerfkitBenchmarker.
> [1] https://beam.apache.org/documentation/io/testing/



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (BEAM-3060) Add performance tests for commonly used file-based I/O PTransforms

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-3060?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16265384#comment-16265384
 ] 

ASF GitHub Bot commented on BEAM-3060:
--

lgajowy commented on a change in pull request #4149: [BEAM-3060] Add Compressed 
TextIOIT
URL: https://github.com/apache/beam/pull/4149#discussion_r152998002
 
 

 ##
 File path: 
sdks/java/io/file-based-io-tests/src/test/java/org/apache/beam/sdk/io/text/TextIOIT.java
 ##
 @@ -83,25 +90,82 @@ private static String appendTimestamp(String 
filenamePrefix) {
 return String.format("%s_%s", filenamePrefix, new Date().getTime());
   }
 
-  @Test
-  public void writeThenReadAll() {
-PCollection testFilenames = pipeline
-.apply("Generate sequence", 
GenerateSequence.from(0).to(numberOfTextLines))
-.apply("Produce text lines", ParDo.of(new 
DeterministicallyConstructTestTextLineFn()))
-.apply("Write content to files", 
TextIO.write().to(filenamePrefix).withOutputFilenames())
-.getPerDestinationOutputFilenames().apply(Values.create());
+  /** IO IT with no compression. */
+  @RunWith(JUnit4.class)
 
 Review comment:
   I double-checked that by running the preCommit job on my machine - those are 
not fired in PreCommit phase. Also, out of curiosity  I investigated a little 
bit the project's mvn structure:
   
   besides the `@RunWith(JUnit.class)` annotation that is required by JUnit, we 
have two mvn plugins that look (scan) for tests:
 - surefire (looks for unit tests and searches for classes with *Test 
suffix)
 - failsafe (looks for integration tests and searches for classes with *IT 
suffix)
   
   As failsafe is not fired in the PreCommit phase, the tests are not invoked. 
Please look at [io parent 
pom](https://github.com/apache/beam/blob/master/sdks/java/io/pom.xml#L77), 
where failsafe plugin is activated only when io-it profile is active. 
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Add performance tests for commonly used file-based I/O PTransforms
> --
>
> Key: BEAM-3060
> URL: https://issues.apache.org/jira/browse/BEAM-3060
> Project: Beam
>  Issue Type: Test
>  Components: sdk-java-core
>Reporter: Chamikara Jayalath
>Assignee: Szymon Nieradka
>
> We recently added a performance testing framework [1] that can be used to do 
> following.
> (1) Execute Beam tests using PerfkitBenchmarker
> (2) Manage Kubernetes-based deployments of data stores.
> (3) Easily publish benchmark results. 
> I think it will be useful to add performance tests for commonly used 
> file-based I/O PTransforms using this framework. I suggest looking into 
> following formats initially.
> (1) AvroIO
> (2) TextIO
> (3) Compressed text using TextIO
> (4) TFRecordIO
> It should be possibly to run these tests for various Beam runners (Direct, 
> Dataflow, Flink, Spark, etc.) and file-systems (GCS, local, HDFS, etc.) 
> easily.
> In the initial version, tests can be made manually triggerable for PRs 
> through Jenkins. Later, we could make some of these tests run periodically 
> and publish benchmark results (to BigQuery) through PerfkitBenchmarker.
> [1] https://beam.apache.org/documentation/io/testing/



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Build failed in Jenkins: beam_PostCommit_Python_Verify #3607

[Apache Jenkins Server]

See 


--
Started by timer
[EnvInject] - Loading node environment variables.
Building remotely on beam3 (beam) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* 
 > +refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/*
 > git rev-parse origin/master^{commit} # timeout=10
Checking out Revision d6345e52e6e8b319456cbad6ce11b05399815930 (origin/master)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f d6345e52e6e8b319456cbad6ce11b05399815930
Commit message: "Update scm to use gitbox.apache.org"
 > git rev-list d6345e52e6e8b319456cbad6ce11b05399815930 # timeout=10
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
[EnvInject] - Executing scripts and injecting environment variables after the 
SCM step.
[EnvInject] - Injecting as environment variables the properties content 
SPARK_LOCAL_IP=127.0.0.1

[EnvInject] - Variables injected successfully.
[beam_PostCommit_Python_Verify] $ /bin/bash -xe 
/tmp/jenkins3043107902306884188.sh
+ bash sdks/python/run_postcommit.sh
bash: sdks/python/run_postcommit.sh: No such file or directory
Build step 'Execute shell' marked build as failure
Not sending mail to unregistered user ajam...@ajamato2016.sea.corp.google.com
Not sending mail to unregistered user c...@google.com
Not sending mail to unregistered user hero...@google.com
Not sending mail to unregistered user git...@nakomis.com
Not sending mail to unregistered user valen...@google.com
Not sending mail to unregistered user 
ke...@kedin-macbookpro.roam.corp.google.com
Not sending mail to unregistered user xumingmi...@gmail.com
Not sending mail to unregistered user k...@google.com
Not sending mail to unregistered user kirpic...@google.com
Not sending mail to unregistered user ekirpic...@gmail.com
ERROR: Unable to send to address: echauc...@gmail.com+jbono...@apache.org
javax.mail.internet.AddressException: Domain contains illegal character in 
string ``echauc...@gmail.com+jbono...@apache.org''
at 
javax.mail.internet.InternetAddress.checkAddress(InternetAddress.java:1269)
at javax.mail.internet.InternetAddress.parse(InternetAddress.java:1091)
at javax.mail.internet.InternetAddress.parse(InternetAddress.java:633)
at javax.mail.internet.InternetAddress.(InternetAddress.java:111)
at hudson.tasks.Mailer.stringToAddress(Mailer.java:206)
at 
jenkins.plugins.mailer.tasks.MimeMessageBuilder.toNormalizedAddress(MimeMessageBuilder.java:309)
at 
jenkins.plugins.mailer.tasks.MimeMessageBuilder.addRecipients(MimeMessageBuilder.java:168)
at 
jenkins.plugins.mailer.tasks.MimeMessageBuilder.addRecipients(MimeMessageBuilder.java:160)
at hudson.tasks.MailSender.createEmptyMail(MailSender.java:389)
at hudson.tasks.MailSender.createFailureMail(MailSender.java:266)
at hudson.tasks.MailSender.createMail(MailSender.java:181)
at hudson.tasks.MailSender.run(MailSender.java:112)
at hudson.tasks.Mailer.perform(Mailer.java:170)
at 
hudson.tasks.BuildStepCompatibilityLayer.perform(BuildStepCompatibilityLayer.java:81)
at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20)
at 
hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:736)
at 
hudson.model.AbstractBuild$AbstractBuildExecution.performAllBuildSteps(AbstractBuild.java:682)
at hudson.model.Build$BuildExecution.post2(Build.java:186)
at 
hudson.model.AbstractBuild$AbstractBuildExecution.post(AbstractBuild.java:627)
at hudson.model.Run.execute(Run.java:1762)
at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
at hudson.model.ResourceController.execute(ResourceController.java:97)
at hudson.model.Executor.run(Executor.java:419)

Build failed in Jenkins: beam_PostCommit_Python_ValidatesRunner_Dataflow #328

[Apache Jenkins Server]

See 


--
Started by timer
[EnvInject] - Loading node environment variables.
Building remotely on beam3 (beam) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* 
 > +refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/*
 > git rev-parse origin/master^{commit} # timeout=10
Checking out Revision d6345e52e6e8b319456cbad6ce11b05399815930 (origin/master)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f d6345e52e6e8b319456cbad6ce11b05399815930
Commit message: "Update scm to use gitbox.apache.org"
 > git rev-list d6345e52e6e8b319456cbad6ce11b05399815930 # timeout=10
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
[EnvInject] - Executing scripts and injecting environment variables after the 
SCM step.
[EnvInject] - Injecting as environment variables the properties content 
SPARK_LOCAL_IP=127.0.0.1

[EnvInject] - Variables injected successfully.
[beam_PostCommit_Python_ValidatesRunner_Dataflow] $ /bin/bash -xe 
/tmp/jenkins5778869543910356601.sh
+ bash sdks/python/run_validatesrunner.sh
bash: sdks/python/run_validatesrunner.sh: No such file or directory
Build step 'Execute shell' marked build as failure

[jira] [Commented] (BEAM-3008) BigtableIO should use ValueProviders

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-3008?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16265314#comment-16265314
 ] 

ASF GitHub Bot commented on BEAM-3008:
--

sduskis commented on a change in pull request #4171: [BEAM-3008] Extends API 
for BigtableIO Read and Write by adding withInstanceId  and withProjectId 
URL: https://github.com/apache/beam/pull/4171#discussion_r152978149
 
 

 ##
 File path: 
sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigtable/BigtableIO.java
 ##
 @@ -78,38 +78,38 @@
  * The Bigtable source returns a set of rows from a single table, returning 
a
  * {@code PCollection}.
  *
- * To configure a Cloud Bigtable source, you must supply a table id and a 
{@link BigtableOptions}
- * or builder configured with the project and other information necessary to 
identify the
- * Bigtable instance. By default, {@link BigtableIO.Read} will read all rows 
in the table. The row
- * range to be read can optionally be restricted using {@link 
BigtableIO.Read#withKeyRange}, and
- * a {@link RowFilter} can be specified using {@link 
BigtableIO.Read#withRowFilter}. For example:
+ * To configure a Cloud Bigtable source, you must supply a table id, a 
project id, an instance
+ * id and optionally a {@link BigtableOptions} to provide more specific 
connection configuration.
+ * By default, {@link BigtableIO.Read} will read all rows in the table. The 
row range to be read
+ * can optionally be restricted using {@link BigtableIO.Read#withKeyRange}, 
and a {@link RowFilter}
+ * can be specified using {@link BigtableIO.Read#withRowFilter}. For example:
  *
  * {@code
- * BigtableOptions.Builder optionsBuilder =
- * new BigtableOptions.Builder()
- * .setProjectId("project")
- * .setInstanceId("instance");
  *
  * Pipeline p = ...;
  *
  * // Scan the entire table.
  * p.apply("read",
  * BigtableIO.read()
  * .withBigtableOptions(optionsBuilder)
 
 Review comment:
   Can you please remove `.withBigtableOptions(optionsBuilder)` for this 
example?


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> BigtableIO should use ValueProviders 
> -
>
> Key: BEAM-3008
> URL: https://issues.apache.org/jira/browse/BEAM-3008
> Project: Beam
>  Issue Type: New Feature
>  Components: sdk-java-gcp
>Reporter: Solomon Duskis
>Assignee: Solomon Duskis
>
> [https://github.com/apache/beam/pull/2057] is an effort towards BigtableIO 
> templatization.  This Issue is a request to get a fully featured template for 
> BigtableIO.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (BEAM-3238) [SQL] Add builder to BeamRecordSqlType

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-3238?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16265289#comment-16265289
 ] 

ASF GitHub Bot commented on BEAM-3238:
--

xumingming commented on issue #4168: [BEAM-3238][SQL] Add 
BeamRecordSqlTypeBuilder
URL: https://github.com/apache/beam/pull/4168#issuecomment-346831152
 
 
   I like this idea!
   
   One minor comment: Can we put the `BeamRecordSqlTypeBuilder` inside 
`BeamRecordSqlType`? it will keep the surface api of 
`org.apache.beam.sdk.extensions.sql` cleaner.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> [SQL] Add builder to BeamRecordSqlType
> --
>
> Key: BEAM-3238
> URL: https://issues.apache.org/jira/browse/BEAM-3238
> Project: Beam
>  Issue Type: Improvement
>  Components: dsl-sql
>Reporter: Anton Kedin
>Assignee: Anton Kedin
>
> Currently it's hard to match field names with types when constructing a 
> BeamRecordSqlType, like 
> [here|https://github.com/apache/beam/blob/39e66e953b0f8e16435acb038cad364acf2b3a57/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/integrationtest/BeamSqlBuiltinFunctionsIntegrationTestBase.java#L64-L71]:
> {code:java}
> BeamRecordSqlType type = BeamRecordSqlType.create(
> Arrays.asList("ts", "c_tinyint", "c_smallint",
> "c_integer", "c_bigint", "c_float", "c_double", "c_decimal",
> "c_tinyint_max", "c_smallint_max", "c_integer_max", "c_bigint_max"),
> Arrays.asList(Types.DATE, Types.TINYINT, Types.SMALLINT,
> Types.INTEGER, Types.BIGINT, Types.FLOAT, Types.DOUBLE, Types.DECIMAL,
> Types.TINYINT, Types.SMALLINT, Types.INTEGER, Types.BIGINT)
> );
> {code}
> It would be much more readable to have a builder, along these lines:
> {code:java}
> BeamRecordSqlType.builder()
>   .withField("f_int", Types.INTEGER)
>   .withStringField("f_str")
>   .build();
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Build failed in Jenkins: beam_PerformanceTests_Python #595

[Apache Jenkins Server]

See 


--
Started by timer
[EnvInject] - Loading node environment variables.
Building remotely on beam7 (beam) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* 
 > +refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/*
 > git rev-parse origin/master^{commit} # timeout=10
Checking out Revision d6345e52e6e8b319456cbad6ce11b05399815930 (origin/master)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f d6345e52e6e8b319456cbad6ce11b05399815930
Commit message: "Update scm to use gitbox.apache.org"
 > git rev-list d6345e52e6e8b319456cbad6ce11b05399815930 # timeout=10
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
[EnvInject] - Executing scripts and injecting environment variables after the 
SCM step.
[EnvInject] - Injecting as environment variables the properties content 
SPARK_LOCAL_IP=127.0.0.1

[EnvInject] - Variables injected successfully.
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins5736435819079351074.sh
+ rm -rf PerfKitBenchmarker
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins7760949034561824259.sh
+ git clone https://github.com/GoogleCloudPlatform/PerfKitBenchmarker.git
Cloning into 'PerfKitBenchmarker'...
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins2981226158658138283.sh
+ pip install --user -r PerfKitBenchmarker/requirements.txt
Requirement already satisfied: absl-py in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 14))
Requirement already satisfied: jinja2>=2.7 in 
/usr/local/lib/python2.7/dist-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 15))
Requirement already satisfied: setuptools in /usr/lib/python2.7/dist-packages 
(from -r PerfKitBenchmarker/requirements.txt (line 16))
Requirement already satisfied: colorlog[windows]==2.6.0 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 17))
Requirement already satisfied: blinker>=1.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 18))
Requirement already satisfied: futures>=3.0.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 19))
Requirement already satisfied: PyYAML==3.12 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 20))
Requirement already satisfied: pint>=0.7 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 21))
Requirement already satisfied: numpy in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 22))
Requirement already satisfied: functools32 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 23))
Requirement already satisfied: contextlib2>=0.5.1 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 24))
Requirement already satisfied: pywinrm in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 25))
Requirement already satisfied: six in 
/home/jenkins/.local/lib/python2.7/site-packages (from absl-py->-r 
PerfKitBenchmarker/requirements.txt (line 14))
Requirement already satisfied: MarkupSafe>=0.23 in 
/usr/local/lib/python2.7/dist-packages (from jinja2>=2.7->-r 
PerfKitBenchmarker/requirements.txt (line 15))
Requirement already satisfied: colorama; extra == "windows" in 
/usr/lib/python2.7/dist-packages (from colorlog[windows]==2.6.0->-r 
PerfKitBenchmarker/requirements.txt (line 17))
Requirement already satisfied: xmltodict in 
/home/jenkins/.local/lib/python2.7/site-packages (from pywinrm->-r 
PerfKitBenchmarker/requirements.txt (line 25))
Requirement already satisfied: requests-ntlm>=0.3.0 in 
/home/jenkins/.local/lib/python2.7/site-packages (from pywinrm->-r 
PerfKitBenchmarker/requirements.txt (line 25))
Requirement already satisfied: requests>=2.9.1 in 
/home/jenkins/.local/lib/python2.7/site-packages (from pywinrm->-r 
PerfKitBenchmarker/requirements.txt (line 25))
Requirement already satisfied: ntlm-auth>=1.0.2 in 
/home/jenkins/.local/lib/python2.7/site-packages (from 
requests-ntlm>=0.3.0->pywinrm->-r PerfKitBenchmarker/requirements.txt (line 25))
Requirement already satisfied: 

Build failed in Jenkins: beam_PerformanceTests_Spark #1042

[Apache Jenkins Server]

See 


--
Started by timer
[EnvInject] - Loading node environment variables.
Building remotely on beam2 (beam) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* 
 > +refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/*
 > git rev-parse origin/master^{commit} # timeout=10
Checking out Revision d6345e52e6e8b319456cbad6ce11b05399815930 (origin/master)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f d6345e52e6e8b319456cbad6ce11b05399815930
Commit message: "Update scm to use gitbox.apache.org"
 > git rev-list d6345e52e6e8b319456cbad6ce11b05399815930 # timeout=10
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
[EnvInject] - Executing scripts and injecting environment variables after the 
SCM step.
[EnvInject] - Injecting as environment variables the properties content 
SPARK_LOCAL_IP=127.0.0.1

[EnvInject] - Variables injected successfully.
[beam_PerformanceTests_Spark] $ /bin/bash -xe /tmp/jenkins3721438551355860794.sh
+ rm -rf PerfKitBenchmarker
[beam_PerformanceTests_Spark] $ /bin/bash -xe /tmp/jenkins6201943027883152253.sh
+ git clone https://github.com/GoogleCloudPlatform/PerfKitBenchmarker.git
Cloning into 'PerfKitBenchmarker'...
[beam_PerformanceTests_Spark] $ /bin/bash -xe /tmp/jenkins6499325939680736724.sh
+ pip install --user -r PerfKitBenchmarker/requirements.txt
Requirement already satisfied: absl-py in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 14))
Requirement already satisfied: jinja2>=2.7 in 
/usr/local/lib/python2.7/dist-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 15))
Requirement already satisfied: setuptools in /usr/lib/python2.7/dist-packages 
(from -r PerfKitBenchmarker/requirements.txt (line 16))
Requirement already satisfied: colorlog[windows]==2.6.0 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 17))
Requirement already satisfied: blinker>=1.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 18))
Requirement already satisfied: futures>=3.0.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 19))
Requirement already satisfied: PyYAML==3.12 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 20))
Requirement already satisfied: pint>=0.7 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 21))
Requirement already satisfied: numpy in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 22))
Requirement already satisfied: functools32 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 23))
Requirement already satisfied: contextlib2>=0.5.1 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 24))
Requirement already satisfied: pywinrm in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 25))
Requirement already satisfied: six in 
/home/jenkins/.local/lib/python2.7/site-packages (from absl-py->-r 
PerfKitBenchmarker/requirements.txt (line 14))
Requirement already satisfied: MarkupSafe in 
/usr/local/lib/python2.7/dist-packages (from jinja2>=2.7->-r 
PerfKitBenchmarker/requirements.txt (line 15))
Requirement already satisfied: colorama; extra == "windows" in 
/usr/lib/python2.7/dist-packages (from colorlog[windows]==2.6.0->-r 
PerfKitBenchmarker/requirements.txt (line 17))
Requirement already satisfied: xmltodict in 
/home/jenkins/.local/lib/python2.7/site-packages (from pywinrm->-r 
PerfKitBenchmarker/requirements.txt (line 25))
Requirement already satisfied: requests-ntlm>=0.3.0 in 
/home/jenkins/.local/lib/python2.7/site-packages (from pywinrm->-r 
PerfKitBenchmarker/requirements.txt (line 25))
Requirement already satisfied: requests>=2.9.1 in 
/home/jenkins/.local/lib/python2.7/site-packages (from pywinrm->-r 
PerfKitBenchmarker/requirements.txt (line 25))
Requirement already satisfied: ntlm-auth>=1.0.2 in 
/home/jenkins/.local/lib/python2.7/site-packages (from 
requests-ntlm>=0.3.0->pywinrm->-r PerfKitBenchmarker/requirements.txt (line 25))
Requirement already satisfied: cryptography>=1.3 in 

[jira] [Commented] (BEAM-3243) multiple anonymous DoFn lead to conflicting names

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-3243?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16265084#comment-16265084
 ] 

ASF GitHub Bot commented on BEAM-3243:
--

rmannibucau opened a new pull request #4172: BEAM-3243 support multiple 
anonymous classes from the same enclosing class in a pipeline
URL: https://github.com/apache/beam/pull/4172
 
 
   Idea is to keep the "number" (suffix) of the anonymous class for anonymous 
dofn to support multiple anonymous dofn in the same pipeline.
   
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> multiple anonymous DoFn lead to conflicting names
> -
>
> Key: BEAM-3243
> URL: https://issues.apache.org/jira/browse/BEAM-3243
> Project: Beam
>  Issue Type: Task
>  Components: sdk-java-core
>Reporter: Romain Manni-Bucau
>Assignee: Romain Manni-Bucau
> Fix For: 2.2.0
>
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (BEAM-3243) multiple anonymous DoFn lead to conflicting names

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-3243?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16265085#comment-16265085
 ] 

ASF GitHub Bot commented on BEAM-3243:
--

GitHub user rmannibucau opened a pull request:

https://github.com/apache/beam/pull/4172

BEAM-3243 support multiple anonymous classes from the same enclosing class 
in a pipeline

Idea is to keep the "number" (suffix) of the anonymous class for anonymous 
dofn to support multiple anonymous dofn in the same pipeline.



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/rmannibucau/incubator-beam 
fb/BEAM-3243_dont-default-to-anonymous-name-for-anonymous-dofn

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/4172.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #4172


commit 76bebb6dc7942b081b386bd5eda7456421b926c7
Author: Romain Manni-Bucau 
Date:   2017-11-24T09:31:04Z

BEAM-3243 default name is not needed by beam in NameUtils so remove it and 
let anonymous class keep their number in their name




> multiple anonymous DoFn lead to conflicting names
> -
>
> Key: BEAM-3243
> URL: https://issues.apache.org/jira/browse/BEAM-3243
> Project: Beam
>  Issue Type: Task
>  Components: sdk-java-core
>Reporter: Romain Manni-Bucau
>Assignee: Romain Manni-Bucau
> Fix For: 2.2.0
>
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (BEAM-3243) multiple anonymous DoFn lead to conflicting names

[Romain Manni-Bucau (JIRA)]

Romain Manni-Bucau created BEAM-3243:


 Summary: multiple anonymous DoFn lead to conflicting names
 Key: BEAM-3243
 URL: https://issues.apache.org/jira/browse/BEAM-3243
 Project: Beam
  Issue Type: Task
  Components: sdk-java-core
Reporter: Romain Manni-Bucau
Assignee: Romain Manni-Bucau
 Fix For: 2.2.0






--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (BEAM-3008) BigtableIO should use ValueProviders

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-3008?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16265065#comment-16265065
 ] 

ASF GitHub Bot commented on BEAM-3008:
--

GitHub user dmytroivanov4206 opened a pull request:

https://github.com/apache/beam/pull/4171

[BEAM-3008] Extends API for BigtableIO Read and Write by adding 
withInstanceId  and withProjectId 

Adds withInstanceId and withProjectId to the BigtableIO Read and Write 
classes, first out of four steps to fix [BEAM-3008] bug.

Follow this checklist to help us incorporate your contribution quickly and 
easily:

 - [x] Make sure there is a [JIRA 
issue](https://issues.apache.org/jira/projects/BEAM/issues/) filed for the 
change (usually before you start working on it).  Trivial changes like typos do 
not require a JIRA issue.  Your pull request should address just this issue, 
without pulling in other changes.
 - [x] Each commit in the pull request should have a meaningful subject 
line and body.
 - [x] Format the pull request title like `[BEAM-XXX] Fixes bug in 
ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA 
issue.
 - [x] Write a pull request description that is detailed enough to 
understand what the pull request does, how, and why.
 - [x] Run `mvn clean verify` to make sure basic checks pass. A more 
thorough check will be performed on your pull request automatically.
 - [ ] If this contribution is large, please file an Apache [Individual 
Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).

---


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/dmytroivanov4206/beam master

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/4171.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #4171


commit 28637766921200f03313088bfc0271453c0e3ae7
Author: Dmytro Ivanov 
Date:   2017-11-24T09:18:14Z

Adds withInstanceId and withProjectId to the BigtableIO Read and Write
classes, first out of four steps to fix [BEAM-3008] bug.




> BigtableIO should use ValueProviders 
> -
>
> Key: BEAM-3008
> URL: https://issues.apache.org/jira/browse/BEAM-3008
> Project: Beam
>  Issue Type: New Feature
>  Components: sdk-java-gcp
>Reporter: Solomon Duskis
>Assignee: Solomon Duskis
>
> [https://github.com/apache/beam/pull/2057] is an effort towards BigtableIO 
> templatization.  This Issue is a request to get a fully featured template for 
> BigtableIO.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (BEAM-3008) BigtableIO should use ValueProviders

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-3008?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16265064#comment-16265064
 ] 

ASF GitHub Bot commented on BEAM-3008:
--

dmytroivanov4206 opened a new pull request #4171: [BEAM-3008] Extends API for 
BigtableIO Read and Write by adding withInstanceId  and withProjectId 
URL: https://github.com/apache/beam/pull/4171
 
 
   Adds withInstanceId and withProjectId to the BigtableIO Read and Write 
classes, first out of four steps to fix [BEAM-3008] bug.
   
   Follow this checklist to help us incorporate your contribution quickly and 
easily:
   
- [x] Make sure there is a [JIRA 
issue](https://issues.apache.org/jira/projects/BEAM/issues/) filed for the 
change (usually before you start working on it).  Trivial changes like typos do 
not require a JIRA issue.  Your pull request should address just this issue, 
without pulling in other changes.
- [x] Each commit in the pull request should have a meaningful subject line 
and body.
- [x] Format the pull request title like `[BEAM-XXX] Fixes bug in 
ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA 
issue.
- [x] Write a pull request description that is detailed enough to 
understand what the pull request does, how, and why.
- [x] Run `mvn clean verify` to make sure basic checks pass. A more 
thorough check will be performed on your pull request automatically.
- [ ] If this contribution is large, please file an Apache [Individual 
Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
   
   ---
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> BigtableIO should use ValueProviders 
> -
>
> Key: BEAM-3008
> URL: https://issues.apache.org/jira/browse/BEAM-3008
> Project: Beam
>  Issue Type: New Feature
>  Components: sdk-java-gcp
>Reporter: Solomon Duskis
>Assignee: Solomon Duskis
>
> [https://github.com/apache/beam/pull/2057] is an effort towards BigtableIO 
> templatization.  This Issue is a request to get a fully featured template for 
> BigtableIO.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Build failed in Jenkins: beam_PostCommit_Python_Verify #3606

[Apache Jenkins Server]

See 


--
Started by timer
[EnvInject] - Loading node environment variables.
Building remotely on beam3 (beam) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* 
 > +refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/*
 > git rev-parse origin/master^{commit} # timeout=10
Checking out Revision d6345e52e6e8b319456cbad6ce11b05399815930 (origin/master)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f d6345e52e6e8b319456cbad6ce11b05399815930
Commit message: "Update scm to use gitbox.apache.org"
 > git rev-list d6345e52e6e8b319456cbad6ce11b05399815930 # timeout=10
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
[EnvInject] - Executing scripts and injecting environment variables after the 
SCM step.
[EnvInject] - Injecting as environment variables the properties content 
SPARK_LOCAL_IP=127.0.0.1

[EnvInject] - Variables injected successfully.
[beam_PostCommit_Python_Verify] $ /bin/bash -xe 
/tmp/jenkins4873058667707421134.sh
+ bash sdks/python/run_postcommit.sh
bash: sdks/python/run_postcommit.sh: No such file or directory
Build step 'Execute shell' marked build as failure
Not sending mail to unregistered user git...@nakomis.com
Not sending mail to unregistered user kirpic...@google.com
Not sending mail to unregistered user k...@google.com
Not sending mail to unregistered user ajam...@ajamato2016.sea.corp.google.com
Not sending mail to unregistered user ekirpic...@gmail.com
Not sending mail to unregistered user 
ke...@kedin-macbookpro.roam.corp.google.com
Not sending mail to unregistered user hero...@google.com
Not sending mail to unregistered user c...@google.com
Not sending mail to unregistered user valen...@google.com
Not sending mail to unregistered user xumingmi...@gmail.com
ERROR: Unable to send to address: echauc...@gmail.com+jbono...@apache.org
javax.mail.internet.AddressException: Domain contains illegal character in 
string ``echauc...@gmail.com+jbono...@apache.org''
at 
javax.mail.internet.InternetAddress.checkAddress(InternetAddress.java:1269)
at javax.mail.internet.InternetAddress.parse(InternetAddress.java:1091)
at javax.mail.internet.InternetAddress.parse(InternetAddress.java:633)
at javax.mail.internet.InternetAddress.(InternetAddress.java:111)
at hudson.tasks.Mailer.stringToAddress(Mailer.java:206)
at 
jenkins.plugins.mailer.tasks.MimeMessageBuilder.toNormalizedAddress(MimeMessageBuilder.java:309)
at 
jenkins.plugins.mailer.tasks.MimeMessageBuilder.addRecipients(MimeMessageBuilder.java:168)
at 
jenkins.plugins.mailer.tasks.MimeMessageBuilder.addRecipients(MimeMessageBuilder.java:160)
at hudson.tasks.MailSender.createEmptyMail(MailSender.java:389)
at hudson.tasks.MailSender.createFailureMail(MailSender.java:266)
at hudson.tasks.MailSender.createMail(MailSender.java:181)
at hudson.tasks.MailSender.run(MailSender.java:112)
at hudson.tasks.Mailer.perform(Mailer.java:170)
at 
hudson.tasks.BuildStepCompatibilityLayer.perform(BuildStepCompatibilityLayer.java:81)
at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20)
at 
hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:736)
at 
hudson.model.AbstractBuild$AbstractBuildExecution.performAllBuildSteps(AbstractBuild.java:682)
at hudson.model.Build$BuildExecution.post2(Build.java:186)
at 
hudson.model.AbstractBuild$AbstractBuildExecution.post(AbstractBuild.java:627)
at hudson.model.Run.execute(Run.java:1762)
at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
at hudson.model.ResourceController.execute(ResourceController.java:97)
at hudson.model.Executor.run(Executor.java:419)

Build failed in Jenkins: beam_PostCommit_Python_ValidatesRunner_Dataflow #327

[Apache Jenkins Server]

See 


--
Started by timer
[EnvInject] - Loading node environment variables.
Building remotely on beam3 (beam) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* 
 > +refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/*
 > git rev-parse origin/master^{commit} # timeout=10
Checking out Revision d6345e52e6e8b319456cbad6ce11b05399815930 (origin/master)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f d6345e52e6e8b319456cbad6ce11b05399815930
Commit message: "Update scm to use gitbox.apache.org"
 > git rev-list d6345e52e6e8b319456cbad6ce11b05399815930 # timeout=10
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
[EnvInject] - Executing scripts and injecting environment variables after the 
SCM step.
[EnvInject] - Injecting as environment variables the properties content 
SPARK_LOCAL_IP=127.0.0.1

[EnvInject] - Variables injected successfully.
[beam_PostCommit_Python_ValidatesRunner_Dataflow] $ /bin/bash -xe 
/tmp/jenkins6081121517994698903.sh
+ bash sdks/python/run_validatesrunner.sh
bash: sdks/python/run_validatesrunner.sh: No such file or directory
Build step 'Execute shell' marked build as failure