See <https://builds.apache.org/job/beam_PerformanceTests_Python/1263/display/redirect?page=changes>
Changes: [rober] Avoid redundant computation when invoking ProcessElements in ParDo [github] Update fn_test.go [rober] De-dup pre & post invoke code in pardo.go [timrobertson100] [BEAM-2661] Adds KuduIO [github] Properly revert log line changes in fn_test.go [rober] Add comparitive benchmark between cached and single use invoker. [github] [BEAM-4852] Only read symbol table when required. [github] Update symbols.go [timrobertson100] [BEAM-2661] KuduIO: Add missing licenses [ehudm] Make with_attributes kwarg optional. [ehudm] Add decode and encode steps to streaming_wordcount [qinyeli] Interactive Beam -- display update [ehudm] Fix PubSubMessageMatcher not acking messages. [ebeach] [BEAM-4862] Fixes bug in Spanner's MutationGroupEncoder by converting [herohde] Add utilities for manipulating proto pipelines [herohde] [BEAM-4636] Refactor Go dataflow runner for more flexible uses [herohde] [BEAM-4813] Switch to translation from model protos to Dataflow v1beta3 [herohde] Add more Go integration tests and fix CoGBK bugs [herohde] [BEAM-4636] Ensure unique filename for compiled worker binaries [herohde] Handle explicit flatten [herohde] Fix Dataflow argument bug [herohde] CR: tweak code and fix comments [aaltay] [BEAM-4751] fix missing pylint3 check for io subpackage (#5916) [pablo] Add GitScm poll trigger for post-commit tests. [relax] Enable schemas for Dataflow runner. [tweise] [BEAM-5023] Send HarnessId in DataClient (#6066) [relax] Fix BigQuery job retry mechanism. [relax] Add exponential backoff on retries. [relax] Apply code-review comment. [tweise] [BEAM-2930] Side inputs are not yet supported in streaming mode. [github] Don't rely on order of elements in a PCollection after GBK in [relax] Properly handle running jobs. [devinduan] Spelling mistakes [relax] Convert BeamSQL to use Schemas. [relax] Deprecate getRowCoder. [relax] Add setSchema to remaining Table objects. [relax] Delete a bunch of code that is no longer used. [relax] Move utilities into schemas.utils package. [relax] Convert Nexmark to use schemas. [relax] remove deprecated Schema.getRowCoder [relax] Apply spotless. [relax] Plumb schema through DoFn. [relax] Add helper setRowSchema to eliminatre boilerplate. [rober] Avoid overwritting user changes to Resolver [rober] Clean up deferedResolver [thw] [BEAM-4842] Update Flink Runner to Flink 1.5.2 [migryz] Add retries for gpg key query logic. [pablo] Fix scheduling for jobs [relax] Add convenience methods for pojo and javabean schema registration. [relax] Address code-review comments. ------------------------------------------ Started by timer [EnvInject] - Loading node environment variables. Building remotely on beam7 (beam) in workspace <https://builds.apache.org/job/beam_PerformanceTests_Python/ws/> > git rev-parse --is-inside-work-tree # timeout=10 Fetching changes from the remote Git repository > git config remote.origin.url https://github.com/apache/beam.git # timeout=10 Fetching upstream changes from https://github.com/apache/beam.git > git --version # timeout=10 > git fetch --tags --progress https://github.com/apache/beam.git > +refs/heads/*:refs/remotes/origin/* > +refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/* > git rev-parse origin/master^{commit} # timeout=10 Checking out Revision f2ae73be3d3d1cb7c3d111505f832aed44bc92eb (origin/master) > git config core.sparsecheckout # timeout=10 > git checkout -f f2ae73be3d3d1cb7c3d111505f832aed44bc92eb Commit message: "Merge pull request #6109: [BEAM-4842] Update Flink Runner to Flink 1.5.2" > git rev-list --no-walk ff95a82e461bd8319d9733be60e75992ba90cd7c # timeout=10 Cleaning workspace > git rev-parse --verify HEAD # timeout=10 Resetting working tree > git reset --hard # timeout=10 > git clean -fdx # timeout=10 [EnvInject] - Executing scripts and injecting environment variables after the SCM step. [EnvInject] - Injecting as environment variables the properties content SPARK_LOCAL_IP=127.0.0.1 [EnvInject] - Variables injected successfully. [beam_PerformanceTests_Python] $ /bin/bash -xe /tmp/jenkins8260919133135875607.sh + rm -rf PerfKitBenchmarker [beam_PerformanceTests_Python] $ /bin/bash -xe /tmp/jenkins3883146302874079411.sh + rm -rf .beam_env [beam_PerformanceTests_Python] $ /bin/bash -xe /tmp/jenkins8573027766605549178.sh + rm -rf .perfkit_env [beam_PerformanceTests_Python] $ /bin/bash -xe /tmp/jenkins7954528295858307459.sh + virtualenv .beam_env --system-site-packages New python executable in <https://builds.apache.org/job/beam_PerformanceTests_Python/ws/.beam_env/bin/python2> Also creating executable in <https://builds.apache.org/job/beam_PerformanceTests_Python/ws/.beam_env/bin/python> Installing setuptools, pkg_resources, pip, wheel...done. Running virtualenv with interpreter /usr/bin/python2 [beam_PerformanceTests_Python] $ /bin/bash -xe /tmp/jenkins117727936914663087.sh + virtualenv .perfkit_env --system-site-packages New python executable in <https://builds.apache.org/job/beam_PerformanceTests_Python/ws/.perfkit_env/bin/python2> Also creating executable in <https://builds.apache.org/job/beam_PerformanceTests_Python/ws/.perfkit_env/bin/python> Installing setuptools, pkg_resources, pip, wheel...done. Running virtualenv with interpreter /usr/bin/python2 [beam_PerformanceTests_Python] $ /bin/bash -xe /tmp/jenkins4530960035642880902.sh + .beam_env/bin/pip install --upgrade setuptools pip Requirement already up-to-date: setuptools in ./.beam_env/lib/python2.7/site-packages (40.0.0) Requirement already up-to-date: pip in ./.beam_env/lib/python2.7/site-packages (18.0) [beam_PerformanceTests_Python] $ /bin/bash -xe /tmp/jenkins5189163966448436943.sh + .perfkit_env/bin/pip install --upgrade setuptools pip Requirement already up-to-date: setuptools in ./.perfkit_env/lib/python2.7/site-packages (40.0.0) Requirement already up-to-date: pip in ./.perfkit_env/lib/python2.7/site-packages (18.0) [beam_PerformanceTests_Python] $ /bin/bash -xe /tmp/jenkins6867585134301396263.sh + git clone https://github.com/GoogleCloudPlatform/PerfKitBenchmarker.git Cloning into 'PerfKitBenchmarker'... [beam_PerformanceTests_Python] $ /bin/bash -xe /tmp/jenkins1463111188888185076.sh + .beam_env/bin/pip install -e 'src/sdks/python/[gcp,test]' Obtaining file://<https://builds.apache.org/job/beam_PerformanceTests_Python/ws/src/sdks/python> Requirement already satisfied: avro<2.0.0,>=1.8.1 in /home/jenkins/.local/lib/python2.7/site-packages (from apache-beam==2.7.0.dev0) (1.8.2) Requirement already satisfied: crcmod<2.0,>=1.7 in /usr/lib/python2.7/dist-packages (from apache-beam==2.7.0.dev0) (1.7) Requirement already satisfied: dill<=0.2.8.2,>=0.2.6 in /home/jenkins/.local/lib/python2.7/site-packages (from apache-beam==2.7.0.dev0) (0.2.8.2) Requirement already satisfied: grpcio<2,>=1.8 in /home/jenkins/.local/lib/python2.7/site-packages (from apache-beam==2.7.0.dev0) (1.13.0) Requirement already satisfied: hdfs<3.0.0,>=2.1.0 in /home/jenkins/.local/lib/python2.7/site-packages (from apache-beam==2.7.0.dev0) (2.1.0) Requirement already satisfied: httplib2<=0.11.3,>=0.8 in /home/jenkins/.local/lib/python2.7/site-packages (from apache-beam==2.7.0.dev0) (0.11.3) Requirement already satisfied: mock<3.0.0,>=1.0.1 in /home/jenkins/.local/lib/python2.7/site-packages (from apache-beam==2.7.0.dev0) (2.0.0) Requirement already satisfied: oauth2client<5,>=2.0.1 in /home/jenkins/.local/lib/python2.7/site-packages (from apache-beam==2.7.0.dev0) (4.1.2) Requirement already satisfied: protobuf<4,>=3.5.0.post1 in /home/jenkins/.local/lib/python2.7/site-packages (from apache-beam==2.7.0.dev0) (3.6.0) Requirement already satisfied: pydot<1.3,>=1.2.0 in /home/jenkins/.local/lib/python2.7/site-packages (from apache-beam==2.7.0.dev0) (1.2.4) Requirement already satisfied: pytz<=2018.4,>=2018.3 in /home/jenkins/.local/lib/python2.7/site-packages (from apache-beam==2.7.0.dev0) (2018.4) Requirement already satisfied: pyyaml<4.0.0,>=3.12 in /usr/local/lib/python2.7/dist-packages (from apache-beam==2.7.0.dev0) (3.12) Requirement already satisfied: pyvcf<0.7.0,>=0.6.8 in /home/jenkins/.local/lib/python2.7/site-packages (from apache-beam==2.7.0.dev0) (0.6.8) Requirement already satisfied: six<1.12,>=1.9 in /usr/local/lib/python2.7/dist-packages (from apache-beam==2.7.0.dev0) (1.11.0) Requirement already satisfied: typing<3.7.0,>=3.6.0 in /home/jenkins/.local/lib/python2.7/site-packages (from apache-beam==2.7.0.dev0) (3.6.4) Requirement already satisfied: futures<4.0.0,>=3.1.1 in /home/jenkins/.local/lib/python2.7/site-packages (from apache-beam==2.7.0.dev0) (3.2.0) Requirement already satisfied: future<1.0.0,>=0.16.0 in /home/jenkins/.local/lib/python2.7/site-packages (from apache-beam==2.7.0.dev0) (0.16.0) Requirement already satisfied: fastavro==0.19.7 in /home/jenkins/.local/lib/python2.7/site-packages (from apache-beam==2.7.0.dev0) (0.19.7) Requirement already satisfied: google-apitools<=0.5.20,>=0.5.18 in /home/jenkins/.local/lib/python2.7/site-packages (from apache-beam==2.7.0.dev0) (0.5.20) Requirement already satisfied: proto-google-cloud-datastore-v1<=0.90.4,>=0.90.0 in /home/jenkins/.local/lib/python2.7/site-packages (from apache-beam==2.7.0.dev0) (0.90.4) Requirement already satisfied: googledatastore==7.0.1 in /home/jenkins/.local/lib/python2.7/site-packages (from apache-beam==2.7.0.dev0) (7.0.1) Requirement already satisfied: google-cloud-pubsub==0.26.0 in /home/jenkins/.local/lib/python2.7/site-packages (from apache-beam==2.7.0.dev0) (0.26.0) Requirement already satisfied: proto-google-cloud-pubsub-v1==0.15.4 in /home/jenkins/.local/lib/python2.7/site-packages (from apache-beam==2.7.0.dev0) (0.15.4) Requirement already satisfied: google-cloud-bigquery==0.25.0 in /home/jenkins/.local/lib/python2.7/site-packages (from apache-beam==2.7.0.dev0) (0.25.0) Requirement already satisfied: nose>=1.3.7 in /home/jenkins/.local/lib/python2.7/site-packages (from apache-beam==2.7.0.dev0) (1.3.7) Collecting numpy<2,>=1.14.3 (from apache-beam==2.7.0.dev0) Using cached https://files.pythonhosted.org/packages/85/51/ba4564ded90e093dbb6adfc3e21f99ae953d9ad56477e1b0d4a93bacf7d3/numpy-1.15.0-cp27-cp27mu-manylinux1_x86_64.whl Requirement already satisfied: pyhamcrest<2.0,>=1.9 in /home/jenkins/.local/lib/python2.7/site-packages (from apache-beam==2.7.0.dev0) (1.9.0) Requirement already satisfied: enum34>=1.0.4 in /usr/local/lib/python2.7/dist-packages (from grpcio<2,>=1.8->apache-beam==2.7.0.dev0) (1.1.6) Requirement already satisfied: docopt in /usr/local/lib/python2.7/dist-packages (from hdfs<3.0.0,>=2.1.0->apache-beam==2.7.0.dev0) (0.6.2) Requirement already satisfied: requests>=2.7.0 in /usr/local/lib/python2.7/dist-packages (from hdfs<3.0.0,>=2.1.0->apache-beam==2.7.0.dev0) (2.18.4) Requirement already satisfied: pbr>=0.11 in /usr/lib/python2.7/dist-packages (from mock<3.0.0,>=1.0.1->apache-beam==2.7.0.dev0) (1.8.0) Requirement already satisfied: funcsigs>=1; python_version < "3.3" in /home/jenkins/.local/lib/python2.7/site-packages (from mock<3.0.0,>=1.0.1->apache-beam==2.7.0.dev0) (1.0.2) Requirement already satisfied: pyasn1>=0.1.7 in /usr/local/lib/python2.7/dist-packages (from oauth2client<5,>=2.0.1->apache-beam==2.7.0.dev0) (0.4.2) Requirement already satisfied: pyasn1-modules>=0.0.5 in /home/jenkins/.local/lib/python2.7/site-packages (from oauth2client<5,>=2.0.1->apache-beam==2.7.0.dev0) (0.2.2) Requirement already satisfied: rsa>=3.1.4 in /home/jenkins/.local/lib/python2.7/site-packages (from oauth2client<5,>=2.0.1->apache-beam==2.7.0.dev0) (3.4.2) Requirement already satisfied: setuptools in ./.beam_env/lib/python2.7/site-packages (from protobuf<4,>=3.5.0.post1->apache-beam==2.7.0.dev0) (40.0.0) Requirement already satisfied: pyparsing>=2.1.4 in /home/jenkins/.local/lib/python2.7/site-packages (from pydot<1.3,>=1.2.0->apache-beam==2.7.0.dev0) (2.2.0) Requirement already satisfied: fasteners>=0.14 in /home/jenkins/.local/lib/python2.7/site-packages (from google-apitools<=0.5.20,>=0.5.18->apache-beam==2.7.0.dev0) (0.14.1) Requirement already satisfied: googleapis-common-protos<2.0dev,>=1.5.2 in /home/jenkins/.local/lib/python2.7/site-packages (from proto-google-cloud-datastore-v1<=0.90.4,>=0.90.0->apache-beam==2.7.0.dev0) (1.5.3) Requirement already satisfied: gapic-google-cloud-pubsub-v1<0.16dev,>=0.15.0 in /home/jenkins/.local/lib/python2.7/site-packages (from google-cloud-pubsub==0.26.0->apache-beam==2.7.0.dev0) (0.15.4) Requirement already satisfied: google-cloud-core<0.26dev,>=0.25.0 in /home/jenkins/.local/lib/python2.7/site-packages (from google-cloud-pubsub==0.26.0->apache-beam==2.7.0.dev0) (0.25.0) Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python2.7/dist-packages (from requests>=2.7.0->hdfs<3.0.0,>=2.1.0->apache-beam==2.7.0.dev0) (2018.4.16) Requirement already satisfied: chardet<3.1.0,>=3.0.2 in /usr/local/lib/python2.7/dist-packages (from requests>=2.7.0->hdfs<3.0.0,>=2.1.0->apache-beam==2.7.0.dev0) (3.0.4) Requirement already satisfied: idna<2.7,>=2.5 in /usr/local/lib/python2.7/dist-packages (from requests>=2.7.0->hdfs<3.0.0,>=2.1.0->apache-beam==2.7.0.dev0) (2.6) Requirement already satisfied: urllib3<1.23,>=1.21.1 in /usr/local/lib/python2.7/dist-packages (from requests>=2.7.0->hdfs<3.0.0,>=2.1.0->apache-beam==2.7.0.dev0) (1.22) Requirement already satisfied: monotonic>=0.1 in /home/jenkins/.local/lib/python2.7/site-packages (from fasteners>=0.14->google-apitools<=0.5.20,>=0.5.18->apache-beam==2.7.0.dev0) (1.5) Requirement already satisfied: google-gax<0.16dev,>=0.15.7 in /home/jenkins/.local/lib/python2.7/site-packages (from gapic-google-cloud-pubsub-v1<0.16dev,>=0.15.0->google-cloud-pubsub==0.26.0->apache-beam==2.7.0.dev0) (0.15.16) Requirement already satisfied: grpc-google-iam-v1<0.12dev,>=0.11.1 in /home/jenkins/.local/lib/python2.7/site-packages (from gapic-google-cloud-pubsub-v1<0.16dev,>=0.15.0->google-cloud-pubsub==0.26.0->apache-beam==2.7.0.dev0) (0.11.4) Requirement already satisfied: google-auth-httplib2 in /home/jenkins/.local/lib/python2.7/site-packages (from google-cloud-core<0.26dev,>=0.25.0->google-cloud-pubsub==0.26.0->apache-beam==2.7.0.dev0) (0.0.3) Requirement already satisfied: google-auth<2.0.0dev,>=0.4.0 in /home/jenkins/.local/lib/python2.7/site-packages (from google-cloud-core<0.26dev,>=0.25.0->google-cloud-pubsub==0.26.0->apache-beam==2.7.0.dev0) (1.5.0) Requirement already satisfied: ply==3.8 in /home/jenkins/.local/lib/python2.7/site-packages (from google-gax<0.16dev,>=0.15.7->gapic-google-cloud-pubsub-v1<0.16dev,>=0.15.0->google-cloud-pubsub==0.26.0->apache-beam==2.7.0.dev0) (3.8) Requirement already satisfied: cachetools>=2.0.0 in /home/jenkins/.local/lib/python2.7/site-packages (from google-auth<2.0.0dev,>=0.4.0->google-cloud-core<0.26dev,>=0.25.0->google-cloud-pubsub==0.26.0->apache-beam==2.7.0.dev0) (2.1.0) Installing collected packages: numpy, apache-beam Found existing installation: numpy 1.13.3 Not uninstalling numpy at /home/jenkins/.local/lib/python2.7/site-packages, outside environment <https://builds.apache.org/job/beam_PerformanceTests_Python/ws/.beam_env> Can't uninstall 'numpy'. No files were found to uninstall. Found existing installation: apache-beam 2.7.0.dev0 Not uninstalling apache-beam at /home/jenkins/jenkins-slave/workspace/beam_PerformanceTests_Compressed_TextIOIT_HDFS/src/sdks/python, outside environment <https://builds.apache.org/job/beam_PerformanceTests_Python/ws/.beam_env> Can't uninstall 'apache-beam'. No files were found to uninstall. Running setup.py develop for apache-beam Successfully installed apache-beam numpy-1.15.0 [beam_PerformanceTests_Python] $ /bin/bash -xe /tmp/jenkins6724119042722330580.sh + .beam_env/bin/python src/sdks/python/setup.py sdist --dist-dir=src/sdks/python/target Traceback (most recent call last): File "src/sdks/python/setup.py", line 46, in <module> PACKAGE_VERSION = get_version() File "src/sdks/python/setup.py", line 41, in get_version exec(open(os.path.normpath('./apache_beam/version.py')).read(), global_names) # pylint: disable=exec-used IOError: [Errno 2] No such file or directory: 'apache_beam/version.py' Build step 'Execute shell' marked build as failure
