See 
<https://builds.apache.org/job/beam_PerformanceTests_Compressed_TextIOIT/76/display/redirect?page=changes>

Changes:

[Pablo] Tracking of time spent reading side inputs, and bytes read in Dataflow.

[Pablo] Fix lint issues

[Pablo] Fixing counter names

[Pablo] Using comments to improve changes

[Pablo] Placing feature behind an experiment flag.

[Pablo] Fixing lint issue

[Pablo] Rebasing

[Pablo] Added cythonization and test

[Pablo] Fixing lint issue

[Pablo] Modifying the experiment flag check to reduce performance impact.

[Pablo] Addressing comments. Profile pending.

[Pablo] Reducing impact - Tracking only Bytes if experiment is active.

[Pablo] Addressing comments

[Pablo] Improved IO Target documentation

[Pablo] Improving documentation

[Pablo] Addressing comments.

[iemejia] Update maven-shade-plugin version to 3.1.0

[iemejia] Update maven-compiler-plugin version to 3.7.0

[iemejia] Update maven-dependency-plugin version to 3.0.2

[iemejia] Update maven-surefire-plugin version to 2.20.1

[iemejia] Update maven-failsafe-plugin version to 2.20.1

[iemejia] Update maven-assembly-plugin version to 3.1.0

[iemejia] Update versions-maven-plugin version to 2.5

[iemejia] Update findbugs-maven-plugin version to 3.0.5

[iemejia] Update license-maven-plugin version to 1.14

[iemejia] Update jacoco-maven-plugin version to 0.7.9

[iemejia] Update dockerfile-maven-plugin version to 1.3.6

[iemejia] Add maven-enforcer-plugin version to pluginManagement

[iemejia] Fix warning on using directly parent.version on Nexmark

[iemejia] Remove warnings on repeated maven-jar-plugin invocation / deps

[iemejia] Remove warning on defining <prerequisites> for non-maven plugin 
project

[iemejia] Update parent POM to version 19

[iemejia] Remove repeated dependency in hadoop-input-format module

------------------------------------------
Started by timer
[EnvInject] - Loading node environment variables.
Building remotely on beam1 (beam) in workspace 
<https://builds.apache.org/job/beam_PerformanceTests_Compressed_TextIOIT/ws/>
 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* 
 > +refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/*
 > git rev-parse origin/master^{commit} # timeout=10
Checking out Revision 21623d892302717a3ad5b3dbc447ebc38268bb29 (origin/master)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 21623d892302717a3ad5b3dbc447ebc38268bb29
Commit message: "Merge pull request #4248 from pabloem/sideinput-io-tracking"
 > git rev-list 4593d9762296f3a012a1e2781d9b9c29e16ffb75 # timeout=10
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
[EnvInject] - Executing scripts and injecting environment variables after the 
SCM step.
[EnvInject] - Injecting as environment variables the properties content 
SPARK_LOCAL_IP=127.0.0.1

[EnvInject] - Variables injected successfully.
[beam_PerformanceTests_Compressed_TextIOIT] $ /bin/bash -xe 
/tmp/jenkins8232151224703208877.sh
+ rm -rf PerfKitBenchmarker
[beam_PerformanceTests_Compressed_TextIOIT] $ /bin/bash -xe 
/tmp/jenkins7696033751867126570.sh
+ rm -rf .env
[beam_PerformanceTests_Compressed_TextIOIT] $ /bin/bash -xe 
/tmp/jenkins3493544444982313687.sh
+ virtualenv .env --system-site-packages
New python executable in 
<https://builds.apache.org/job/beam_PerformanceTests_Compressed_TextIOIT/ws/.env/bin/python>
Installing setuptools, pip, wheel...done.
[beam_PerformanceTests_Compressed_TextIOIT] $ /bin/bash -xe 
/tmp/jenkins2383081597567248049.sh
+ git clone https://github.com/GoogleCloudPlatform/PerfKitBenchmarker.git
Cloning into 'PerfKitBenchmarker'...
[beam_PerformanceTests_Compressed_TextIOIT] $ /bin/bash -xe 
/tmp/jenkins5805455928864201787.sh
+ .env/bin/pip install -r PerfKitBenchmarker/requirements.txt
Requirement already satisfied: absl-py in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 14))
Requirement already satisfied: jinja2>=2.7 in 
/usr/local/lib/python2.7/dist-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 15))
Requirement already satisfied: setuptools in ./.env/lib/python2.7/site-packages 
(from -r PerfKitBenchmarker/requirements.txt (line 16))
Requirement already satisfied: colorlog[windows]==2.6.0 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 17))
Requirement already satisfied: blinker>=1.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 18))
Requirement already satisfied: futures>=3.0.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 19))
Requirement already satisfied: PyYAML==3.12 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 20))
Requirement already satisfied: pint>=0.7 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 21))
Requirement already satisfied: numpy in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 22))
Requirement already satisfied: functools32 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 23))
Requirement already satisfied: contextlib2>=0.5.1 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 24))
Requirement already satisfied: pywinrm in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 25))
Requirement already satisfied: six in 
/home/jenkins/.local/lib/python2.7/site-packages (from absl-py->-r 
PerfKitBenchmarker/requirements.txt (line 14))
Requirement already satisfied: MarkupSafe in 
/usr/local/lib/python2.7/dist-packages (from jinja2>=2.7->-r 
PerfKitBenchmarker/requirements.txt (line 15))
Requirement already satisfied: colorama; extra == "windows" in 
/usr/lib/python2.7/dist-packages (from colorlog[windows]==2.6.0->-r 
PerfKitBenchmarker/requirements.txt (line 17))
Requirement already satisfied: xmltodict in 
/home/jenkins/.local/lib/python2.7/site-packages (from pywinrm->-r 
PerfKitBenchmarker/requirements.txt (line 25))
Requirement already satisfied: requests-ntlm>=0.3.0 in 
/home/jenkins/.local/lib/python2.7/site-packages (from pywinrm->-r 
PerfKitBenchmarker/requirements.txt (line 25))
Requirement already satisfied: requests>=2.9.1 in 
/home/jenkins/.local/lib/python2.7/site-packages (from pywinrm->-r 
PerfKitBenchmarker/requirements.txt (line 25))
Requirement already satisfied: ntlm-auth>=1.0.2 in 
/home/jenkins/.local/lib/python2.7/site-packages (from 
requests-ntlm>=0.3.0->pywinrm->-r PerfKitBenchmarker/requirements.txt (line 25))
Requirement already satisfied: cryptography>=1.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from 
requests-ntlm>=0.3.0->pywinrm->-r PerfKitBenchmarker/requirements.txt (line 25))
Requirement already satisfied: idna<2.6,>=2.5 in 
/home/jenkins/.local/lib/python2.7/site-packages (from 
requests>=2.9.1->pywinrm->-r PerfKitBenchmarker/requirements.txt (line 25))
Requirement already satisfied: urllib3<1.22,>=1.21.1 in 
/home/jenkins/.local/lib/python2.7/site-packages (from 
requests>=2.9.1->pywinrm->-r PerfKitBenchmarker/requirements.txt (line 25))
Requirement already satisfied: chardet<3.1.0,>=3.0.2 in 
/home/jenkins/.local/lib/python2.7/site-packages (from 
requests>=2.9.1->pywinrm->-r PerfKitBenchmarker/requirements.txt (line 25))
Requirement already satisfied: certifi>=2017.4.17 in 
/home/jenkins/.local/lib/python2.7/site-packages (from 
requests>=2.9.1->pywinrm->-r PerfKitBenchmarker/requirements.txt (line 25))
Requirement already satisfied: cffi>=1.7; platform_python_implementation != 
"PyPy" in /home/jenkins/.local/lib/python2.7/site-packages (from 
cryptography>=1.3->requests-ntlm>=0.3.0->pywinrm->-r 
PerfKitBenchmarker/requirements.txt (line 25))
Requirement already satisfied: enum34; python_version < "3" in 
/home/jenkins/.local/lib/python2.7/site-packages (from 
cryptography>=1.3->requests-ntlm>=0.3.0->pywinrm->-r 
PerfKitBenchmarker/requirements.txt (line 25))
Requirement already satisfied: asn1crypto>=0.21.0 in 
/home/jenkins/.local/lib/python2.7/site-packages (from 
cryptography>=1.3->requests-ntlm>=0.3.0->pywinrm->-r 
PerfKitBenchmarker/requirements.txt (line 25))
Requirement already satisfied: ipaddress; python_version < "3" in 
/usr/local/lib/python2.7/dist-packages (from 
cryptography>=1.3->requests-ntlm>=0.3.0->pywinrm->-r 
PerfKitBenchmarker/requirements.txt (line 25))
Requirement already satisfied: pycparser in 
/home/jenkins/.local/lib/python2.7/site-packages (from cffi>=1.7; 
platform_python_implementation != 
"PyPy"->cryptography>=1.3->requests-ntlm>=0.3.0->pywinrm->-r 
PerfKitBenchmarker/requirements.txt (line 25))
[beam_PerformanceTests_Compressed_TextIOIT] $ /bin/bash -xe 
/tmp/jenkins2983202131740478758.sh
+ .env/bin/pip install -e 'src/sdks/python/[gcp,test]'
Obtaining 
file://<https://builds.apache.org/job/beam_PerformanceTests_Compressed_TextIOIT/ws/src/sdks/python>
Requirement already satisfied: avro<2.0.0,>=1.8.1 in 
/home/jenkins/.local/lib/python2.7/site-packages (from apache-beam==2.4.0.dev0)
Requirement already satisfied: crcmod<2.0,>=1.7 in 
/home/jenkins/.local/lib/python2.7/site-packages (from apache-beam==2.4.0.dev0)
Requirement already satisfied: dill==0.2.6 in 
/home/jenkins/.local/lib/python2.7/site-packages (from apache-beam==2.4.0.dev0)
Requirement already satisfied: grpcio<2,>=1.0 in 
/home/jenkins/.local/lib/python2.7/site-packages (from apache-beam==2.4.0.dev0)
Requirement already satisfied: httplib2<0.10,>=0.8 in 
/home/jenkins/.local/lib/python2.7/site-packages (from apache-beam==2.4.0.dev0)
Requirement already satisfied: mock<3.0.0,>=1.0.1 in 
/home/jenkins/.local/lib/python2.7/site-packages (from apache-beam==2.4.0.dev0)
Requirement already satisfied: oauth2client<5,>=2.0.1 in 
/home/jenkins/.local/lib/python2.7/site-packages (from apache-beam==2.4.0.dev0)
Requirement already satisfied: protobuf<4,>=3.5.0.post1 in 
/home/jenkins/.local/lib/python2.7/site-packages (from apache-beam==2.4.0.dev0)
Requirement already satisfied: pyyaml<4.0.0,>=3.12 in 
/home/jenkins/.local/lib/python2.7/site-packages (from apache-beam==2.4.0.dev0)
Requirement already satisfied: pyvcf<0.7.0,>=0.6.8 in 
/home/jenkins/.local/lib/python2.7/site-packages (from apache-beam==2.4.0.dev0)
Requirement already satisfied: six<1.12,>=1.9 in 
/home/jenkins/.local/lib/python2.7/site-packages (from apache-beam==2.4.0.dev0)
Requirement already satisfied: typing<3.7.0,>=3.6.0 in 
/home/jenkins/.local/lib/python2.7/site-packages (from apache-beam==2.4.0.dev0)
Requirement already satisfied: futures<4.0.0,>=3.1.1 in 
/home/jenkins/.local/lib/python2.7/site-packages (from apache-beam==2.4.0.dev0)
Requirement already satisfied: hdfs3<0.4.0,>=0.3.0 in 
/home/jenkins/.local/lib/python2.7/site-packages (from apache-beam==2.4.0.dev0)
Requirement already satisfied: google-apitools<=0.5.20,>=0.5.18 in 
/home/jenkins/.local/lib/python2.7/site-packages (from apache-beam==2.4.0.dev0)
Requirement already satisfied: proto-google-cloud-datastore-v1<=0.90.4,>=0.90.0 
in /home/jenkins/.local/lib/python2.7/site-packages (from 
apache-beam==2.4.0.dev0)
Requirement already satisfied: googledatastore==7.0.1 in 
/home/jenkins/.local/lib/python2.7/site-packages (from apache-beam==2.4.0.dev0)
Requirement already satisfied: google-cloud-pubsub==0.26.0 in 
/home/jenkins/.local/lib/python2.7/site-packages (from apache-beam==2.4.0.dev0)
Requirement already satisfied: google-cloud-bigquery==0.25.0 in 
/home/jenkins/.local/lib/python2.7/site-packages (from apache-beam==2.4.0.dev0)
Requirement already satisfied: pyhamcrest<2.0,>=1.9 in 
/home/jenkins/.local/lib/python2.7/site-packages (from apache-beam==2.4.0.dev0)
Requirement already satisfied: enum34>=1.0.4 in 
/home/jenkins/.local/lib/python2.7/site-packages (from 
grpcio<2,>=1.0->apache-beam==2.4.0.dev0)
Requirement already satisfied: funcsigs>=1; python_version < "3.3" in 
/home/jenkins/.local/lib/python2.7/site-packages (from 
mock<3.0.0,>=1.0.1->apache-beam==2.4.0.dev0)
Requirement already satisfied: pbr>=0.11 in 
/home/jenkins/.local/lib/python2.7/site-packages (from 
mock<3.0.0,>=1.0.1->apache-beam==2.4.0.dev0)
Requirement already satisfied: pyasn1>=0.1.7 in 
/home/jenkins/.local/lib/python2.7/site-packages (from 
oauth2client<5,>=2.0.1->apache-beam==2.4.0.dev0)
Requirement already satisfied: pyasn1-modules>=0.0.5 in 
/home/jenkins/.local/lib/python2.7/site-packages (from 
oauth2client<5,>=2.0.1->apache-beam==2.4.0.dev0)
Requirement already satisfied: rsa>=3.1.4 in 
/home/jenkins/.local/lib/python2.7/site-packages (from 
oauth2client<5,>=2.0.1->apache-beam==2.4.0.dev0)
Requirement already satisfied: setuptools in ./.env/lib/python2.7/site-packages 
(from protobuf<4,>=3.5.0.post1->apache-beam==2.4.0.dev0)
Requirement already satisfied: fasteners>=0.14 in 
/home/jenkins/.local/lib/python2.7/site-packages (from 
google-apitools<=0.5.20,>=0.5.18->apache-beam==2.4.0.dev0)
Requirement already satisfied: googleapis-common-protos<2.0dev,>=1.5.2 in 
/home/jenkins/.local/lib/python2.7/site-packages (from 
proto-google-cloud-datastore-v1<=0.90.4,>=0.90.0->apache-beam==2.4.0.dev0)
Requirement already satisfied: google-cloud-core<0.26dev,>=0.25.0 in 
/home/jenkins/.local/lib/python2.7/site-packages (from 
google-cloud-pubsub==0.26.0->apache-beam==2.4.0.dev0)
Requirement already satisfied: gapic-google-cloud-pubsub-v1<0.16dev,>=0.15.0 in 
/home/jenkins/.local/lib/python2.7/site-packages (from 
google-cloud-pubsub==0.26.0->apache-beam==2.4.0.dev0)
Requirement already satisfied: monotonic>=0.1 in 
/home/jenkins/.local/lib/python2.7/site-packages (from 
fasteners>=0.14->google-apitools<=0.5.20,>=0.5.18->apache-beam==2.4.0.dev0)
Requirement already satisfied: google-auth<2.0.0dev,>=0.4.0 in 
/home/jenkins/.local/lib/python2.7/site-packages (from 
google-cloud-core<0.26dev,>=0.25.0->google-cloud-pubsub==0.26.0->apache-beam==2.4.0.dev0)
Requirement already satisfied: google-auth-httplib2 in 
/home/jenkins/.local/lib/python2.7/site-packages (from 
google-cloud-core<0.26dev,>=0.25.0->google-cloud-pubsub==0.26.0->apache-beam==2.4.0.dev0)
Requirement already satisfied: google-gax<0.16dev,>=0.15.7 in 
/home/jenkins/.local/lib/python2.7/site-packages (from 
gapic-google-cloud-pubsub-v1<0.16dev,>=0.15.0->google-cloud-pubsub==0.26.0->apache-beam==2.4.0.dev0)
Requirement already satisfied: 
proto-google-cloud-pubsub-v1[grpc]<0.16dev,>=0.15.4 in 
/home/jenkins/.local/lib/python2.7/site-packages (from 
gapic-google-cloud-pubsub-v1<0.16dev,>=0.15.0->google-cloud-pubsub==0.26.0->apache-beam==2.4.0.dev0)
Requirement already satisfied: grpc-google-iam-v1<0.12dev,>=0.11.1 in 
/home/jenkins/.local/lib/python2.7/site-packages (from 
gapic-google-cloud-pubsub-v1<0.16dev,>=0.15.0->google-cloud-pubsub==0.26.0->apache-beam==2.4.0.dev0)
Requirement already satisfied: cachetools>=2.0.0 in 
/home/jenkins/.local/lib/python2.7/site-packages (from 
google-auth<2.0.0dev,>=0.4.0->google-cloud-core<0.26dev,>=0.25.0->google-cloud-pubsub==0.26.0->apache-beam==2.4.0.dev0)
Requirement already satisfied: future<0.17dev,>=0.16.0 in 
/home/jenkins/.local/lib/python2.7/site-packages (from 
google-gax<0.16dev,>=0.15.7->gapic-google-cloud-pubsub-v1<0.16dev,>=0.15.0->google-cloud-pubsub==0.26.0->apache-beam==2.4.0.dev0)
Requirement already satisfied: ply==3.8 in 
/home/jenkins/.local/lib/python2.7/site-packages (from 
google-gax<0.16dev,>=0.15.7->gapic-google-cloud-pubsub-v1<0.16dev,>=0.15.0->google-cloud-pubsub==0.26.0->apache-beam==2.4.0.dev0)
Requirement already satisfied: requests<3.0dev,>=2.13.0 in 
/home/jenkins/.local/lib/python2.7/site-packages (from 
google-gax<0.16dev,>=0.15.7->gapic-google-cloud-pubsub-v1<0.16dev,>=0.15.0->google-cloud-pubsub==0.26.0->apache-beam==2.4.0.dev0)
Requirement already satisfied: idna<2.6,>=2.5 in 
/home/jenkins/.local/lib/python2.7/site-packages (from 
requests<3.0dev,>=2.13.0->google-gax<0.16dev,>=0.15.7->gapic-google-cloud-pubsub-v1<0.16dev,>=0.15.0->google-cloud-pubsub==0.26.0->apache-beam==2.4.0.dev0)
Requirement already satisfied: urllib3<1.22,>=1.21.1 in 
/home/jenkins/.local/lib/python2.7/site-packages (from 
requests<3.0dev,>=2.13.0->google-gax<0.16dev,>=0.15.7->gapic-google-cloud-pubsub-v1<0.16dev,>=0.15.0->google-cloud-pubsub==0.26.0->apache-beam==2.4.0.dev0)
Requirement already satisfied: chardet<3.1.0,>=3.0.2 in 
/home/jenkins/.local/lib/python2.7/site-packages (from 
requests<3.0dev,>=2.13.0->google-gax<0.16dev,>=0.15.7->gapic-google-cloud-pubsub-v1<0.16dev,>=0.15.0->google-cloud-pubsub==0.26.0->apache-beam==2.4.0.dev0)
Requirement already satisfied: certifi>=2017.4.17 in 
/home/jenkins/.local/lib/python2.7/site-packages (from 
requests<3.0dev,>=2.13.0->google-gax<0.16dev,>=0.15.7->gapic-google-cloud-pubsub-v1<0.16dev,>=0.15.0->google-cloud-pubsub==0.26.0->apache-beam==2.4.0.dev0)
Installing collected packages: apache-beam
  Found existing installation: apache-beam 2.4.0.dev0
    Not uninstalling apache-beam at 
/home/jenkins/jenkins-slave/workspace/beam_PerformanceTests_TextIOIT/src/sdks/python,
 outside environment 
<https://builds.apache.org/job/beam_PerformanceTests_Compressed_TextIOIT/ws/.env>
  Running setup.py develop for apache-beam
Successfully installed apache-beam
[beam_PerformanceTests_Compressed_TextIOIT] $ /bin/bash -xe 
/tmp/jenkins1970665503440264762.sh
+ .env/bin/python PerfKitBenchmarker/pkb.py --project=apache-beam-testing 
--dpb_log_level=INFO --maven_binary=/home/jenkins/tools/maven/latest/bin/mvn 
--bigquery_table=beam_performance.compressed_textioit_pkb_results 
--official=true --benchmarks=beam_integration_benchmark --beam_it_timeout=1200 
--beam_it_profile=io-it --beam_prebuilt=true --beam_sdk=java 
--beam_it_module=sdks/java/io/file-based-io-tests 
--beam_it_class=org.apache.beam.sdk.io.text.TextIOIT 
'--beam_it_options=[--project=apache-beam-testing,--tempRoot=gs://temp-storage-for-perf-tests,--numberOfRecords=1000000,--filenamePrefix=gs://temp-storage-for-perf-tests/beam_PerformanceTests_Compressed_TextIOIT/76/,--compressionType=GZIP]'
 '--beam_extra_mvn_properties=[filesystem=gcs]'
Traceback (most recent call last):
  File "PerfKitBenchmarker/pkb.py", line 21, in <module>
    sys.exit(Main())
  File 
"<https://builds.apache.org/job/beam_PerformanceTests_Compressed_TextIOIT/ws/PerfKitBenchmarker/perfkitbenchmarker/pkb.py";,>
 line 869, in Main
    SetUpPKB()
  File 
"<https://builds.apache.org/job/beam_PerformanceTests_Compressed_TextIOIT/ws/PerfKitBenchmarker/perfkitbenchmarker/pkb.py";,>
 line 740, in SetUpPKB
    vm_util.GenTempDir()
  File 
"<https://builds.apache.org/job/beam_PerformanceTests_Compressed_TextIOIT/ws/PerfKitBenchmarker/perfkitbenchmarker/vm_util.py";,>
 line 121, in GenTempDir
    temp_dir.CreateTemporaryDirectories()
  File 
"<https://builds.apache.org/job/beam_PerformanceTests_Compressed_TextIOIT/ws/PerfKitBenchmarker/perfkitbenchmarker/temp_dir.py";,>
 line 62, in CreateTemporaryDirectories
    os.makedirs(path)
  File 
"<https://builds.apache.org/job/beam_PerformanceTests_Compressed_TextIOIT/ws/.env/lib/python2.7/os.py";,>
 line 157, in makedirs
    mkdir(name, mode)
OSError: [Errno 13] Permission denied: '/tmp/perfkitbenchmarker/runs/fc872370'
Build step 'Execute shell' marked build as failure

Reply via email to