I checked out PerfKitBenchmarker. It seems we can get rid of the error either by defining a default value for runner_profile_override :
+def AddRunnerProfileMvnArgument(service_type, mvn_command, + runner_profile_override): Or passing None in the places where AddRunnerProfileMvnArgument is called. On Sat, Jul 1, 2017 at 9:19 AM, Ted Yu <[email protected]> wrote: > > File "<https://builds.apache.org/job/beam_PerformanceTests_ > > Python/ws/PerfKitBenchmarker/perfkitbenchmarker/beam_benchma > rk_helper.py",> > > line 136, in InitializeBeamRepo > > mvn_command) > > TypeError: AddRunnerProfileMvnArgument() takes exactly 3 arguments (2 > > given) > > Looks like 3rd parameter is needed to initialize repo. > > beam_benchmark_helper.py doesn't seem to be in source repo, though. > > > On Fri, Jun 30, 2017 at 10:55 PM, Kenneth Knowles <[email protected]> > wrote: > >> This build has been failing for two days. Does anyone have any insights? >> >> On Fri, Jun 30, 2017 at 10:50 PM, Apache Jenkins Server < >> [email protected]> wrote: >> >> > See <https://builds.apache.org/job/beam_PerformanceTests_ >> > Python/47/display/redirect> >> > >> > ------------------------------------------ >> > Started by timer >> > [EnvInject] - Loading node environment variables. >> > Building remotely on beam3 (beam) in workspace < >> https://builds.apache.org/ >> > job/beam_PerformanceTests_Python/ws/> >> > > git rev-parse --is-inside-work-tree # timeout=10 >> > Fetching changes from the remote Git repository >> > > git config remote.origin.url https://github.com/apache/beam.git # >> > timeout=10 >> > Fetching upstream changes from https://github.com/apache/beam.git >> > > git --version # timeout=10 >> > > git fetch --tags --progress https://github.com/apache/beam.git >> > +refs/heads/*:refs/remotes/origin/* +refs/pull/${ghprbPullId}/*: >> > refs/remotes/origin/pr/${ghprbPullId}/* >> > > git rev-parse origin/master^{commit} # timeout=10 >> > Checking out Revision 0e429b33ff85eba08da5018c9febd0b99b44f720 >> > (origin/master) >> > > git config core.sparsecheckout # timeout=10 >> > > git checkout -f 0e429b33ff85eba08da5018c9febd0b99b44f720 >> > > git rev-list 0e429b33ff85eba08da5018c9febd0b99b44f720 # timeout=10 >> > Cleaning workspace >> > > git rev-parse --verify HEAD # timeout=10 >> > Resetting working tree >> > > git reset --hard # timeout=10 >> > > git clean -fdx # timeout=10 >> > [EnvInject] - Executing scripts and injecting environment variables >> after >> > the SCM step. >> > [EnvInject] - Injecting as environment variables the properties content >> > SPARK_LOCAL_IP=127.0.0.1 >> > >> > [EnvInject] - Variables injected successfully. >> > [beam_PerformanceTests_Python] $ /bin/bash -xe /tmp/ >> > hudson1915973589795310141.sh >> > + rm -rf PerfKitBenchmarker >> > [beam_PerformanceTests_Python] $ /bin/bash -xe /tmp/ >> > hudson7830718553943537745.sh >> > + git clone https://github.com/GoogleCloudPlatform/PerfKitBenchmarker. >> git >> > Cloning into 'PerfKitBenchmarker'... >> > [beam_PerformanceTests_Python] $ /bin/bash -xe /tmp/ >> > hudson7887504202385599721.sh >> > + pip install --user -r PerfKitBenchmarker/requirements.txt >> > Requirement already satisfied (use --upgrade to upgrade): >> > python-gflags==3.1.1 in /home/jenkins/.local/lib/pytho >> n2.7/site-packages >> > (from -r PerfKitBenchmarker/requirements.txt (line 14)) >> > Requirement already satisfied (use --upgrade to upgrade): jinja2>=2.7 in >> > /usr/local/lib/python2.7/dist-packages (from -r >> PerfKitBenchmarker/requirements.txt >> > (line 15)) >> > Requirement already satisfied (use --upgrade to upgrade): setuptools in >> > /usr/lib/python2.7/dist-packages (from -r >> PerfKitBenchmarker/requirements.txt >> > (line 16)) >> > Requirement already satisfied (use --upgrade to upgrade): >> > colorlog[windows]==2.6.0 in /home/jenkins/.local/lib/pytho >> n2.7/site-packages >> > (from -r PerfKitBenchmarker/requirements.txt (line 17)) >> > Installing extra requirements: 'windows' >> > Requirement already satisfied (use --upgrade to upgrade): blinker>=1.3 >> in >> > /home/jenkins/.local/lib/python2.7/site-packages (from -r >> > PerfKitBenchmarker/requirements.txt (line 18)) >> > Requirement already satisfied (use --upgrade to upgrade): futures>=3.0.3 >> > in /home/jenkins/.local/lib/python2.7/site-packages (from -r >> > PerfKitBenchmarker/requirements.txt (line 19)) >> > Requirement already satisfied (use --upgrade to upgrade): PyYAML==3.12 >> in >> > /home/jenkins/.local/lib/python2.7/site-packages (from -r >> > PerfKitBenchmarker/requirements.txt (line 20)) >> > Requirement already satisfied (use --upgrade to upgrade): pint>=0.7 in >> > /home/jenkins/.local/lib/python2.7/site-packages (from -r >> > PerfKitBenchmarker/requirements.txt (line 21)) >> > Requirement already satisfied (use --upgrade to upgrade): numpy in >> > /home/jenkins/.local/lib/python2.7/site-packages (from -r >> > PerfKitBenchmarker/requirements.txt (line 22)) >> > Requirement already satisfied (use --upgrade to upgrade): functools32 in >> > /home/jenkins/.local/lib/python2.7/site-packages (from -r >> > PerfKitBenchmarker/requirements.txt (line 23)) >> > Requirement already satisfied (use --upgrade to upgrade): >> > contextlib2>=0.5.1 in /home/jenkins/.local/lib/python2.7/site-packages >> > (from -r PerfKitBenchmarker/requirements.txt (line 24)) >> > Cleaning up... >> > [beam_PerformanceTests_Python] $ /bin/bash -xe /tmp/ >> > hudson8072560625143652403.sh >> > + pip install --user -e 'sdks/python/[gcp,test]' >> > Obtaining file://<https://builds.apache.org/job/beam_PerformanceTests_ >> > Python/ws/sdks/python> >> > Running setup.py (path:<https://builds.apache. >> > org/job/beam_PerformanceTests_Python/ws/sdks/python/setup.py)> egg_info >> > for package from file://<https://builds.apache. >> > org/job/beam_PerformanceTests_Python/ws/sdks/python> >> > <https://builds.apache.org/job/beam_PerformanceTests_ >> > Python/ws/sdks/python/setup.py>:66: UserWarning: You are using version >> > 1.5.4 of pip. However, version 7.0.0 is recommended. >> > _PIP_VERSION, REQUIRED_PIP_VERSION >> > no previously-included directories found matching 'doc/.build' >> > >> > Installed <https://builds.apache.org/job/beam_PerformanceTests_ >> > Python/ws/sdks/python/nose-1.3.7-py2.7.egg> >> > >> > warning: no files found matching 'README.md' >> > warning: no files found matching 'NOTICE' >> > warning: no files found matching 'LICENSE' >> > Installing extra requirements: 'gcp,test' >> > Requirement already satisfied (use --upgrade to upgrade): >> > avro>=1.8.1,<2.0.0 in /home/jenkins/.local/lib/python2.7/site-packages >> > (from apache-beam==2.1.0.dev) >> > Requirement already satisfied (use --upgrade to upgrade): >> crcmod>=1.7,<2.0 >> > in /home/jenkins/.local/lib/python2.7/site-packages (from >> > apache-beam==2.1.0.dev) >> > Requirement already satisfied (use --upgrade to upgrade): dill==0.2.6 in >> > /home/jenkins/.local/lib/python2.7/site-packages (from >> > apache-beam==2.1.0.dev) >> > Requirement already satisfied (use --upgrade to upgrade): >> grpcio>=1.0,<2.0 >> > in /home/jenkins/.local/lib/python2.7/site-packages (from >> > apache-beam==2.1.0.dev) >> > Requirement already satisfied (use --upgrade to upgrade): >> > httplib2>=0.8,<0.10 in /home/jenkins/.local/lib/python2.7/site-packages >> > (from apache-beam==2.1.0.dev) >> > Requirement already satisfied (use --upgrade to upgrade): >> > mock>=1.0.1,<3.0.0 in /home/jenkins/.local/lib/python2.7/site-packages >> > (from apache-beam==2.1.0.dev) >> > Requirement already satisfied (use --upgrade to upgrade): >> > oauth2client>=2.0.1,<4.0.0 in /home/jenkins/.local/lib/pytho >> n2.7/site-packages >> > (from apache-beam==2.1.0.dev) >> > Requirement already satisfied (use --upgrade to upgrade): >> > protobuf>=3.2.0,<=3.3.0 in /home/jenkins/.local/lib/pytho >> n2.7/site-packages >> > (from apache-beam==2.1.0.dev) >> > Requirement already satisfied (use --upgrade to upgrade): >> > pyyaml>=3.12,<4.0.0 in /home/jenkins/.local/lib/python2.7/site-packages >> > (from apache-beam==2.1.0.dev) >> > Requirement already satisfied (use --upgrade to upgrade): >> > pyhamcrest>=1.9,<2.0 in /home/jenkins/.local/lib/pytho >> n2.7/site-packages >> > (from apache-beam==2.1.0.dev) >> > Requirement already satisfied (use --upgrade to upgrade): six>=1.9 in >> > /home/jenkins/.local/lib/python2.7/site-packages (from >> > apache-beam==2.1.0.dev) >> > Requirement already satisfied (use --upgrade to upgrade): >> > google-apitools>=0.5.10,<=0.5.11 in /home/jenkins/.local/lib/pytho >> n2.7/site-packages >> > (from apache-beam==2.1.0.dev) >> > Requirement already satisfied (use --upgrade to upgrade): >> > proto-google-cloud-datastore-v1>=0.90.0,<=0.90.4 in >> > /home/jenkins/.local/lib/python2.7/site-packages (from >> > apache-beam==2.1.0.dev) >> > Requirement already satisfied (use --upgrade to upgrade): >> > googledatastore==7.0.1 in /home/jenkins/.local/lib/pytho >> n2.7/site-packages >> > (from apache-beam==2.1.0.dev) >> > Requirement already satisfied (use --upgrade to upgrade): >> > google-cloud-pubsub==0.25.0 in /home/jenkins/.local/lib/pytho >> n2.7/site-packages >> > (from apache-beam==2.1.0.dev) >> > Requirement already satisfied (use --upgrade to upgrade): >> > google-cloud-bigquery>=0.23.0,<0.25.0 in /home/jenkins/.local/lib/pytho >> n2.7/site-packages >> > (from apache-beam==2.1.0.dev) >> > Requirement already satisfied (use --upgrade to upgrade): futures>=2.2.0 >> > in /home/jenkins/.local/lib/python2.7/site-packages (from >> > grpcio>=1.0,<2.0->apache-beam==2.1.0.dev) >> > Requirement already satisfied (use --upgrade to upgrade): enum34>=1.0.4 >> in >> > /home/jenkins/.local/lib/python2.7/site-packages (from >> > grpcio>=1.0,<2.0->apache-beam==2.1.0.dev) >> > Requirement already satisfied (use --upgrade to upgrade): pyasn1>=0.1.7 >> in >> > /home/jenkins/.local/lib/python2.7/site-packages (from >> > oauth2client>=2.0.1,<4.0.0->apache-beam==2.1.0.dev) >> > Requirement already satisfied (use --upgrade to upgrade): >> > pyasn1-modules>=0.0.5 in /home/jenkins/.local/lib/pytho >> n2.7/site-packages >> > (from oauth2client>=2.0.1,<4.0.0->apache-beam==2.1.0.dev) >> > Requirement already satisfied (use --upgrade to upgrade): rsa>=3.1.4 in >> > /home/jenkins/.local/lib/python2.7/site-packages (from >> > oauth2client>=2.0.1,<4.0.0->apache-beam==2.1.0.dev) >> > Requirement already satisfied (use --upgrade to upgrade): setuptools in >> > /usr/lib/python2.7/dist-packages (from protobuf>=3.2.0,<=3.3.0-> >> > apache-beam==2.1.0.dev) >> > Requirement already satisfied (use --upgrade to upgrade): >> > googleapis-common-protos>=1.5.2,<2.0dev in >> /home/jenkins/.local/lib/python2.7/site-packages >> > (from proto-google-cloud-datastore-v1>=0.90.0,<=0.90.4->apache- >> > beam==2.1.0.dev) >> > Installing collected packages: apache-beam >> > Running setup.py develop for apache-beam >> > <https://builds.apache.org/job/beam_PerformanceTests_ >> > Python/ws/sdks/python/setup.py>:66: UserWarning: You are using version >> > 1.5.4 of pip. However, version 7.0.0 is recommended. >> > _PIP_VERSION, REQUIRED_PIP_VERSION >> > >> > warning: no files found matching 'README.md' >> > warning: no files found matching 'NOTICE' >> > warning: no files found matching 'LICENSE' >> > Creating /home/jenkins/.local/lib/python2.7/site-packages/apache- >> beam.egg-link >> > (link to .) >> > apache-beam 2.1.0.dev is already the active version in >> easy-install.pth >> > >> > Installed <https://builds.apache.org/job/beam_PerformanceTests_ >> > Python/ws/sdks/python> >> > Successfully installed apache-beam >> > Cleaning up... >> > [beam_PerformanceTests_Python] $ /bin/bash -xe /tmp/ >> > hudson7511664461243177645.sh >> > + python PerfKitBenchmarker/pkb.py --project=apache-beam-testing >> > --dpb_log_level=INFO --maven_binary=/home/jenkins/t >> ools/maven/latest/bin/mvn >> > --bigquery_table=beam_performance.pkb_results --official=true >> > --beam_sdk=python --benchmarks=beam_integration_benchmark >> > --beam_it_args=--project=apache-beam-testing,--staging_ >> > location=gs://temp-storage-for-end-to-end-tests/staging- >> > it,--temp_location=gs://temp-storage-for-end-to-end-tests/ >> > temp-it,--output=gs://temp-storage-for-end-to-end-tests/py- >> it-cloud/output >> > WARNING:root:File resource loader root perfkitbenchmarker/data/ycsb is >> not >> > a directory. >> > 2017-07-01 05:50:07,346 10b71d64 MainThread INFO Verbose logging to: >> > /tmp/perfkitbenchmarker/runs/10b71d64/pkb.log >> > 2017-07-01 05:50:07,347 10b71d64 MainThread INFO PerfKitBenchmarker >> > version: v1.12.0-53-ga07b524 >> > 2017-07-01 05:50:07,347 10b71d64 MainThread INFO Flag values: >> > --beam_sdk=python >> > --beam_it_args=--project=apache-beam-testing,--staging_ >> > location=gs://temp-storage-for-end-to-end-tests/staging- >> > it,--temp_location=gs://temp-storage-for-end-to-end-tests/ >> > temp-it,--output=gs://temp-storage-for-end-to-end-tests/py- >> it-cloud/output >> > --maven_binary=/home/jenkins/tools/maven/latest/bin/mvn >> > --project=apache-beam-testing >> > --bigquery_table=beam_performance.pkb_results >> > --official >> > --dpb_log_level=INFO >> > --benchmarks=beam_integration_benchmark >> > 2017-07-01 05:50:07,785 10b71d64 MainThread WARNING The key "flags" was >> > not in the default config, but was in user overrides. This may indicate >> a >> > typo. >> > 2017-07-01 05:50:08,046 10b71d64 MainThread >> beam_integration_benchmark(1/1) >> > INFO Provisioning resources for benchmark beam_integration_benchmark >> > 2017-07-01 05:50:08,049 10b71d64 MainThread >> beam_integration_benchmark(1/1) >> > INFO Preparing benchmark beam_integration_benchmark >> > 2017-07-01 05:50:08,049 10b71d64 MainThread >> beam_integration_benchmark(1/1) >> > INFO Running: git clone https://github.com/apache/beam.git >> > 2017-07-01 05:50:13,212 10b71d64 MainThread >> beam_integration_benchmark(1/1) >> > ERROR Error during benchmark beam_integration_benchmark >> > Traceback (most recent call last): >> > File "<https://builds.apache.org/job/beam_PerformanceTests_ >> > Python/ws/PerfKitBenchmarker/perfkitbenchmarker/pkb.py",> line 555, in >> > RunBenchmark >> > DoPreparePhase(spec, detailed_timer) >> > File "<https://builds.apache.org/job/beam_PerformanceTests_ >> > Python/ws/PerfKitBenchmarker/perfkitbenchmarker/pkb.py",> line 450, in >> > DoPreparePhase >> > spec.BenchmarkPrepare(spec) >> > File "<https://builds.apache.org/job/beam_PerformanceTests_ >> > Python/ws/PerfKitBenchmarker/perfkitbenchmarker/linux_ >> > benchmarks/beam_integration_benchmark.py",> line 91, in Prepare >> > beam_benchmark_helper.InitializeBeamRepo(benchmark_spec) >> > File "<https://builds.apache.org/job/beam_PerformanceTests_ >> > Python/ws/PerfKitBenchmarker/perfkitbenchmarker/beam_benchma >> rk_helper.py",> >> > line 136, in InitializeBeamRepo >> > mvn_command) >> > TypeError: AddRunnerProfileMvnArgument() takes exactly 3 arguments (2 >> > given) >> > 2017-07-01 05:50:13,214 10b71d64 MainThread >> beam_integration_benchmark(1/1) >> > ERROR Benchmark 1/1 beam_integration_benchmark (UID: >> > beam_integration_benchmark0) failed. Execution will continue. >> > 2017-07-01 05:50:13,237 10b71d64 MainThread INFO Benchmark run >> > statuses: >> > --------------------------------------------------------------- >> > Name UID Status >> > --------------------------------------------------------------- >> > beam_integration_benchmark beam_integration_benchmark0 FAILED >> > --------------------------------------------------------------- >> > Success rate: 0.00% (0/1) >> > 2017-07-01 05:50:13,237 10b71d64 MainThread INFO Complete logs can >> be >> > found at: /tmp/perfkitbenchmarker/runs/10b71d64/pkb.log >> > 2017-07-01 05:50:13,237 10b71d64 MainThread INFO Completion statuses >> > can be found at: /tmp/perfkitbenchmarker/runs/ >> > 10b71d64/completion_statuses.json >> > Build step 'Execute shell' marked build as failure >> > Not sending mail to unregistered user [email protected] >> > Not sending mail to unregistered user [email protected] >> > Not sending mail to unregistered user [email protected] >> > >> > >
