See <https://builds.apache.org/job/beam_PreCommit_Portable_Python_Cron/1798/display/redirect?page=changes>
Changes: [michal.walenia] [BEAM-9258] Add integration test for Cloud DLP ------------------------------------------ [...truncated 86.24 KB...] Stored in directory: /home/jenkins/.cache/pip/wheels/b9/23/19/54774416627a49bc596b602ac356ae76d22c95394e37702cab Successfully built apache-beam Installing collected packages: crcmod, dill, fastavro, docopt, certifi, idna, chardet, urllib3, requests, hdfs, httplib2, pbr, funcsigs, mock, numpy, pymongo, pyasn1, rsa, pyasn1-modules, oauth2client, pyparsing, pydot, python-dateutil, pytz, avro, pyvcf, pyarrow, typing-extensions, cachetools, monotonic, fasteners, google-apitools, google-auth, googleapis-common-protos, google-api-core, google-cloud-core, google-cloud-datastore, grpc-google-iam-v1, google-cloud-pubsub, google-resumable-media, google-cloud-bigquery, google-cloud-bigtable, google-cloud-spanner, grpcio-gcp, google-cloud-dlp, google-cloud-language, google-cloud-videointelligence, google-cloud-vision, proto-google-cloud-datastore-v1, googledatastore, freezegun, nose, nose-xunitmp, pandas, parameterized, pyhamcrest, pyyaml, requests-mock, tenacity, atomicwrites, packaging, attrs, wcwidth, more-itertools, pytest, pytest-forked, apipkg, execnet, pytest-xdist, pytest-timeout, apache-beam > Task :runners:flink:1.9:job-server:shadowJar > Task :runners:flink:1.9:job-server-container:copyDockerfileDependencies > Task :sdks:python:test-suites:portable:py2:installGcpTest Successfully installed apache-beam-2.20.0.dev0 apipkg-1.5 atomicwrites-1.3.0 attrs-19.3.0 avro-1.9.2 cachetools-3.1.1 certifi-2019.11.28 chardet-3.0.4 crcmod-1.7 dill-0.3.1.1 docopt-0.6.2 execnet-1.7.1 fastavro-0.21.24 fasteners-0.15 freezegun-0.3.15 funcsigs-1.0.2 google-api-core-1.16.0 google-apitools-0.5.28 google-auth-1.11.2 google-cloud-bigquery-1.24.0 google-cloud-bigtable-1.0.0 google-cloud-core-1.3.0 google-cloud-datastore-1.7.4 google-cloud-dlp-0.13.0 google-cloud-language-1.3.0 google-cloud-pubsub-1.0.2 google-cloud-spanner-1.13.0 google-cloud-videointelligence-1.13.0 google-cloud-vision-0.42.0 google-resumable-media-0.5.0 googleapis-common-protos-1.51.0 googledatastore-7.0.2 grpc-google-iam-v1-0.12.3 grpcio-gcp-0.2.2 hdfs-2.5.8 httplib2-0.12.0 idna-2.9 mock-2.0.0 monotonic-1.5 more-itertools-5.0.0 nose-1.3.7 nose-xunitmp-0.4.1 numpy-1.16.6 oauth2client-3.0.0 packaging-20.1 pandas-0.24.2 parameterized-0.7.1 pbr-5.4.4 proto-google-cloud-datastore-v1-0.90.4 pyarrow-0.15.1 pyasn1-0.4.8 pyasn1-modules-0.2.8 pydot-1.4.1 pyhamcrest-1.10.1 pymongo-3.10.1 pyparsing-2.4.6 pytest-4.6.9 pytest-forked-1.1.3 pytest-timeout-1.3.4 pytest-xdist-1.31.0 python-dateutil-2.8.1 pytz-2019.3 pyvcf-0.6.8 pyyaml-5.3 requests-2.23.0 requests-mock-1.7.0 rsa-4.0 tenacity-5.1.5 typing-extensions-3.7.4.1 urllib3-1.25.8 wcwidth-0.1.8 > Task :runners:flink:1.9:job-server-container:dockerPrepare > Task :sdks:go:resolveBuildDependencies Resolving google.golang.org/api: commit='386d4e5f4f92f86e6aec85985761bba4b938a2d5', urls=[https://code.googlesource.com/google-api-go-client] > Task :runners:flink:1.9:job-server-container:docker > Task :sdks:go:resolveBuildDependencies Resolving google.golang.org/genproto: commit='2b5a72b8730b0b16380010cfe5286c42108d88e7', urls=[https://github.com/google/go-genproto] Resolving google.golang.org/grpc: commit='7646b5360d049a7ca31e9133315db43456f39e2e', urls=[https://github.com/grpc/grpc-go] Resolving cached github.com/coreos/etcd: commit='11214aa33bf5a47d3d9d8dafe0f6b97237dfe921', urls=[https://github.com/coreos/etcd.git, [email protected]:coreos/etcd.git] Resolving cached github.com/coreos/etcd: commit='11214aa33bf5a47d3d9d8dafe0f6b97237dfe921', urls=[https://github.com/coreos/etcd.git, [email protected]:coreos/etcd.git] > Task :sdks:go:installDependencies > Task :sdks:go:buildLinuxAmd64 > Task :sdks:go:goBuild > Task :sdks:python:container:resolveBuildDependencies Resolving ./github.com/apache/beam/sdks/go@<https://builds.apache.org/job/beam_PreCommit_Portable_Python_Cron/ws/src/sdks/go> > Task :sdks:python:container:installDependencies > Task :sdks:python:container:buildDarwinAmd64 > Task :sdks:python:container:buildLinuxAmd64 > Task :sdks:python:container:goBuild > Task :sdks:python:container:py2:copyLauncherDependencies > Task :sdks:python:container:py2:dockerPrepare > Task :sdks:python:container:py2:docker The command '/bin/sh -c pip install -v /opt/apache/beam/tars/apache-beam.tar.gz[gcp] && rm -rf /root/.cache/pip' returned a non-zero code: 1 > Task :sdks:python:container:py2:docker FAILED > Task :sdks:python:test-suites:portable:py2:portableWordCountBatch <https://builds.apache.org/job/beam_PreCommit_Portable_Python_Cron/ws/src/build/gradleenv/1866363813/local/lib/python2.7/site-packages/apache_beam/__init__.py>:82: UserWarning: You are using Apache Beam with Python 2. New releases of Apache Beam will soon support Python 3 only. 'You are using Apache Beam with Python 2. ' WARNING:root:Make sure that locally built Python SDK docker image has Python 2.7 interpreter. INFO:root:Using Python SDK docker image: apache/beam_python2.7_sdk:2.20.0.dev. If the image is not available at local, we will try to pull from hub.docker.com INFO:apache_beam.runners.portability.fn_api_runner_transforms:==================== <function lift_combiners at 0x7fc855871d70> ==================== WARNING:apache_beam.utils.subprocess_server:Starting service with ['docker' 'run' '-v' '/usr/bin/docker:/bin/docker' '-v' '/var/run/docker.sock:/var/run/docker.sock' '--network=host' 'apache/beam_flink1.9_job_server:latest' '--job-host' 'localhost' '--job-port' '59883' '--artifact-port' '40663' '--expansion-port' '59727'] WARNING:root:Waiting for grpc channel to be ready at localhost:59883. WARNING:root:Waiting for grpc channel to be ready at localhost:59883. WARNING:root:Waiting for grpc channel to be ready at localhost:59883. WARNING:root:Waiting for grpc channel to be ready at localhost:59883. WARNING:root:Waiting for grpc channel to be ready at localhost:59883. [main] INFO org.apache.beam.runners.fnexecution.jobsubmission.JobServerDriver - ArtifactStagingService started on localhost:40663 [main] INFO org.apache.beam.runners.fnexecution.jobsubmission.JobServerDriver - Java ExpansionService started on localhost:59727 [main] INFO org.apache.beam.runners.fnexecution.jobsubmission.JobServerDriver - JobService started on localhost:59883 WARNING:root:Waiting for grpc channel to be ready at localhost:59883. [grpc-default-executor-0] INFO org.apache.beam.runners.flink.FlinkJobInvoker - Invoking job BeamApp-root-0227182732-75930c08_ac2d7d52-5534-4f64-9a6f-22d5ac943666 with pipeline runner org.apache.beam.runners.flink.FlinkPipelineRunner@37177f94 [grpc-default-executor-0] INFO org.apache.beam.runners.fnexecution.jobsubmission.JobInvocation - Starting job invocation BeamApp-root-0227182732-75930c08_ac2d7d52-5534-4f64-9a6f-22d5ac943666 [flink-runner-job-invoker] INFO org.apache.beam.runners.flink.FlinkPipelineRunner - Translating pipeline to Flink program. INFO:apache_beam.runners.portability.portable_runner:Job state changed to STOPPED INFO:apache_beam.runners.portability.portable_runner:Job state changed to STARTING INFO:apache_beam.runners.portability.portable_runner:Job state changed to RUNNING [flink-runner-job-invoker] INFO org.apache.beam.runners.flink.FlinkExecutionEnvironments - Creating a Batch Execution Environment. [flink-runner-job-invoker] INFO org.apache.flink.api.java.ExecutionEnvironment - The job has 0 registered types and 0 default Kryo serializers [flink-runner-job-invoker] INFO org.apache.flink.runtime.minicluster.MiniCluster - Starting Flink Mini Cluster [flink-runner-job-invoker] INFO org.apache.flink.runtime.minicluster.MiniCluster - Starting Metrics Registry [flink-runner-job-invoker] INFO org.apache.flink.runtime.metrics.MetricRegistryImpl - No metrics reporter configured, no metrics will be exposed/reported. [flink-runner-job-invoker] INFO org.apache.flink.runtime.minicluster.MiniCluster - Starting RPC Service(s) [flink-akka.actor.default-dispatcher-2] INFO akka.event.slf4j.Slf4jLogger - Slf4jLogger started [flink-runner-job-invoker] INFO org.apache.flink.runtime.rpc.akka.AkkaRpcServiceUtils - Trying to start actor system at :0 [flink-metrics-2] INFO akka.event.slf4j.Slf4jLogger - Slf4jLogger started [flink-metrics-2] INFO akka.remote.Remoting - Starting remoting [flink-metrics-2] INFO akka.remote.Remoting - Remoting started; listening on addresses :[akka.tcp://[email protected]:39805] [flink-runner-job-invoker] INFO org.apache.flink.runtime.rpc.akka.AkkaRpcServiceUtils - Actor system started at akka.tcp://[email protected]:39805 [flink-runner-job-invoker] INFO org.apache.flink.runtime.rpc.akka.AkkaRpcService - Starting RPC endpoint for org.apache.flink.runtime.metrics.dump.MetricQueryService at akka://flink-metrics/user/MetricQueryService . [flink-runner-job-invoker] INFO org.apache.flink.runtime.minicluster.MiniCluster - Starting high-availability services [flink-runner-job-invoker] ERROR org.apache.beam.runners.fnexecution.jobsubmission.JobInvocation - Error during job invocation BeamApp-root-0227182732-75930c08_ac2d7d52-5534-4f64-9a6f-22d5ac943666. java.io.IOException: Could not create storage directory for BLOB store in '/tmp'. at org.apache.flink.runtime.blob.BlobUtils.initLocalStorageDirectory(BlobUtils.java:173) at org.apache.flink.runtime.blob.BlobServer.<init>(BlobServer.java:140) at org.apache.flink.runtime.minicluster.MiniCluster.start(MiniCluster.java:300) at org.apache.flink.client.LocalExecutor.createJobExecutorService(LocalExecutor.java:143) at org.apache.flink.client.LocalExecutor.start(LocalExecutor.java:118) at org.apache.flink.client.LocalExecutor.executePlan(LocalExecutor.java:203) at org.apache.flink.api.java.LocalEnvironment.execute(LocalEnvironment.java:91) at org.apache.beam.runners.flink.FlinkBatchPortablePipelineTranslator$BatchTranslationContext.execute(FlinkBatchPortablePipelineTranslator.java:194) at org.apache.beam.runners.flink.FlinkPipelineRunner.runPipelineWithTranslator(FlinkPipelineRunner.java:116) at org.apache.beam.runners.flink.FlinkPipelineRunner.run(FlinkPipelineRunner.java:84) at org.apache.beam.runners.fnexecution.jobsubmission.JobInvocation.runPipeline(JobInvocation.java:83) at org.apache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:125) at org.apache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:57) at org.apache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:78) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) ERROR:root:java.io.IOException: Could not create storage directory for BLOB store in '/tmp'. [flink-runner-job-invoker] INFO org.apache.beam.runners.fnexecution.artifact.AbstractArtifactRetrievalService - Manifest at /tmp/beam-artifact-staging/job_46e90f98-8257-458e-8499-a2f8cd790f41/MANIFEST has 1 artifact locations [flink-runner-job-invoker] INFO org.apache.beam.runners.fnexecution.artifact.BeamFileSystemArtifactStagingService - Removed dir /tmp/beam-artifact-staging/job_46e90f98-8257-458e-8499-a2f8cd790f41/ INFO:apache_beam.runners.portability.portable_runner:Job state changed to FAILED Traceback (most recent call last): File "/usr/lib/python2.7/runpy.py", line 174, in _run_module_as_main "__main__", fname, loader, pkg_name) File "/usr/lib/python2.7/runpy.py", line 72, in _run_code exec code in run_globals File "<https://builds.apache.org/job/beam_PreCommit_Portable_Python_Cron/ws/src/build/gradleenv/1866363813/lib/python2.7/site-packages/apache_beam/examples/wordcount.py";,> line 142, in <module> run() File "<https://builds.apache.org/job/beam_PreCommit_Portable_Python_Cron/ws/src/build/gradleenv/1866363813/lib/python2.7/site-packages/apache_beam/examples/wordcount.py";,> line 122, in run result.wait_until_finish() File "<https://builds.apache.org/job/beam_PreCommit_Portable_Python_Cron/ws/src/build/gradleenv/1866363813/local/lib/python2.7/site-packages/apache_beam/runners/portability/portable_runner.py";,> line 528, in wait_until_finish (self._job_id, self._state, self._last_error_message())) RuntimeError: Pipeline BeamApp-root-0227182732-75930c08_ac2d7d52-5534-4f64-9a6f-22d5ac943666 failed in state FAILED: java.io.IOException: Could not create storage directory for BLOB store in '/tmp'. > Task :sdks:python:test-suites:portable:py2:portableWordCountBatch FAILED > Task :sdks:python:test-suites:portable:py2:portableWordCountStreaming <https://builds.apache.org/job/beam_PreCommit_Portable_Python_Cron/ws/src/build/gradleenv/1866363813/local/lib/python2.7/site-packages/apache_beam/__init__.py>:82: UserWarning: You are using Apache Beam with Python 2. New releases of Apache Beam will soon support Python 3 only. 'You are using Apache Beam with Python 2. ' WARNING:root:Make sure that locally built Python SDK docker image has Python 2.7 interpreter. INFO:root:Using Python SDK docker image: apache/beam_python2.7_sdk:2.20.0.dev. If the image is not available at local, we will try to pull from hub.docker.com WARNING:apache_beam.utils.subprocess_server:Starting service with ['docker' 'run' '-v' '/usr/bin/docker:/bin/docker' '-v' '/var/run/docker.sock:/var/run/docker.sock' '--network=host' 'apache/beam_flink1.9_job_server:latest' '--job-host' 'localhost' '--job-port' '49089' '--artifact-port' '46261' '--expansion-port' '33891'] WARNING:root:Waiting for grpc channel to be ready at localhost:49089. WARNING:root:Waiting for grpc channel to be ready at localhost:49089. [main] INFO org.apache.beam.runners.fnexecution.jobsubmission.JobServerDriver - ArtifactStagingService started on localhost:46261 [main] INFO org.apache.beam.runners.fnexecution.jobsubmission.JobServerDriver - Java ExpansionService started on localhost:33891 [main] INFO org.apache.beam.runners.fnexecution.jobsubmission.JobServerDriver - JobService started on localhost:49089 WARNING:root:Waiting for grpc channel to be ready at localhost:49089. WARNING:root:Waiting for grpc channel to be ready at localhost:49089. Traceback (most recent call last): File "/usr/lib/python2.7/runpy.py", line 174, in _run_module_as_main "__main__", fname, loader, pkg_name) File "/usr/lib/python2.7/runpy.py", line 72, in _run_code exec code in run_globals File "<https://builds.apache.org/job/beam_PreCommit_Portable_Python_Cron/ws/src/build/gradleenv/1866363813/lib/python2.7/site-packages/apache_beam/examples/wordcount.py";,> line 142, in <module> run() File "<https://builds.apache.org/job/beam_PreCommit_Portable_Python_Cron/ws/src/build/gradleenv/1866363813/lib/python2.7/site-packages/apache_beam/examples/wordcount.py";,> line 121, in run result = p.run() File "<https://builds.apache.org/job/beam_PreCommit_Portable_Python_Cron/ws/src/build/gradleenv/1866363813/local/lib/python2.7/site-packages/apache_beam/pipeline.py";,> line 483, in run self._options).run(False) File "<https://builds.apache.org/job/beam_PreCommit_Portable_Python_Cron/ws/src/build/gradleenv/1866363813/local/lib/python2.7/site-packages/apache_beam/pipeline.py";,> line 496, in run return self.runner.run_pipeline(self, self._options) File "<https://builds.apache.org/job/beam_PreCommit_Portable_Python_Cron/ws/src/build/gradleenv/1866363813/local/lib/python2.7/site-packages/apache_beam/runners/portability/portable_runner.py";,> line 384, in run_pipeline job_service_plan.submit(proto_pipeline) File "<https://builds.apache.org/job/beam_PreCommit_Portable_Python_Cron/ws/src/build/gradleenv/1866363813/local/lib/python2.7/site-packages/apache_beam/runners/portability/portable_runner.py";,> line 105, in submit prepare_response.staging_session_token) File "<https://builds.apache.org/job/beam_PreCommit_Portable_Python_Cron/ws/src/build/gradleenv/1866363813/local/lib/python2.7/site-packages/apache_beam/runners/portability/portable_runner.py";,> line 190, in stage staging_location='') File "<https://builds.apache.org/job/beam_PreCommit_Portable_Python_Cron/ws/src/build/gradleenv/1866363813/local/lib/python2.7/site-packages/apache_beam/runners/portability/stager.py";,> line 150, in stage_job_resources temp_dir = temp_dir or tempfile.mkdtemp() File "/usr/lib/python2.7/tempfile.py", line 339, in mkdtemp _os.mkdir(file, 0700) OSError: [Errno 28] No space left on device: '/tmp/tmpQ7wfFd' > Task :sdks:python:test-suites:portable:py2:portableWordCountStreaming FAILED FAILURE: Build completed with 3 failures. 1: Task failed with an exception. ----------- * What went wrong: Execution failed for task ':sdks:python:container:py2:docker'. > Process 'command 'docker'' finished with non-zero exit value 1 * Try: Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. Run with --scan to get full insights. ============================================================================== 2: Task failed with an exception. ----------- * What went wrong: Execution failed for task ':sdks:python:test-suites:portable:py2:portableWordCountBatch'. > Process 'command 'sh'' finished with non-zero exit value 1 * Try: Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. Run with --scan to get full insights. ============================================================================== 3: Task failed with an exception. ----------- * What went wrong: Execution failed for task ':sdks:python:test-suites:portable:py2:portableWordCountStreaming'. > Process 'command 'sh'' finished with non-zero exit value 1 * Try: Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. Run with --scan to get full insights. ============================================================================== * Get more help at https://help.gradle.org Deprecated Gradle features were used in this build, making it incompatible with Gradle 6.0. Use '--warning-mode all' to show the individual deprecation warnings. See https://docs.gradle.org/5.2.1/userguide/command_line_interface.html#sec:command_line_warnings BUILD FAILED in 9m 58s 82 actionable tasks: 63 executed, 17 from cache, 2 up-to-date A build scan cannot be produced as an error occurred spooling the build data. Please report this problem via https://gradle.com/scans/help/plugin and include the following via copy/paste: ---------- Gradle version: 5.2.1 Plugin version: 2.3 java.lang.IllegalStateException: java.io.IOException: No space left on device at com.gradle.scan.a.e.b.flush(SourceFile:233) at com.gradle.scan.a.e.b.close(SourceFile:244) at com.gradle.scan.a.e.c.close(SourceFile:66) at com.gradle.scan.plugin.internal.n.a(SourceFile:78) at com.gradle.scan.plugin.internal.e.c.c.b(SourceFile:125) at com.gradle.scan.plugin.internal.e.c.c.a(SourceFile:117) at com.gradle.scan.plugin.internal.o.a$a.a(SourceFile:30) at com.gradle.scan.plugin.internal.o.a$a.a(SourceFile:19) at com.gradle.scan.plugin.internal.o.a.c(SourceFile:60) Caused by: java.io.IOException: No space left on device at com.gradle.scan.a.e.b.flush(SourceFile:231) ... 8 more ---------- FAILURE: Build failed with an exception. * What went wrong: Failed to notify root build lifecycle listener. > Could not update > /home/jenkins/.gradle/caches/5.2.1/file-changes/last-build.bin > Could not update > <https://builds.apache.org/job/beam_PreCommit_Portable_Python_Cron/ws/src/.gradle/5.2.1/fileChanges/last-build.bin> * Try: Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. Run with --scan to get full insights. * Get more help at https://help.gradle.org Deprecated Gradle features were used in this build, making it incompatible with Gradle 6.0. Use '--warning-mode all' to show the individual deprecation warnings. See https://docs.gradle.org/5.2.1/userguide/command_line_interface.html#sec:command_line_warnings BUILD FAILED in 9m 58s FAILURE: Build failed with an exception. * What went wrong: Could not write cache value to '/home/jenkins/.gradle/daemon/5.2.1/registry.bin'. * Try: Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. Run with --scan to get full insights. * Get more help at https://help.gradle.org Build step 'Invoke Gradle script' changed build result to FAILURE Build step 'Invoke Gradle script' marked build as failure --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
