See <https://builds.apache.org/job/beam_PerformanceTests_TextIOIT/488/display/redirect?page=changes>
Changes: [github] Explictly delcare globals defined elsewhere [mairbek] Introduced SpannerWriteResult that [mairbek] Addressed comments [mairbek] Happy checkstyle [github] Adding a microbenchmark for side input iterables. (#5294) [apilloud] Enable githubCommitNotifier for post commits [tgroh] Migrate the `portable` subpackage to Portability ------------------------------------------ [...truncated 50.35 KB...] > Task :beam-sdks-java-build-tools:packageTests > Task :beam-runners-apex:processResources NO-SOURCE > Task :beam-runners-apex:processTestResources > Task :beam-model-pipeline:compileJava FROM-CACHE > Task :beam-model-pipeline:processResources > Task :beam-model-pipeline:classes > Task :beam-sdks-java-extensions-sql:generateFmppSources > Task :beam-sdks-java-build-tools:install > Task :beam-sdks-python:setupVirtualenv done. Running virtualenv with interpreter /usr/bin/python2 Requirement already up-to-date: tox==3.0.0 in /home/jenkins/.local/lib/python2.7/site-packages (3.0.0) Collecting grpcio-tools==1.3.5 > Task :beam-model-pipeline:shadowJar > Task :beam-sdks-java-core:compileJava FROM-CACHE > Task :beam-sdks-java-core:processResources > Task :beam-sdks-java-core:classes > Task :beam-sdks-java-extensions-sql:compileJavacc Java Compiler Compiler Version 4.0 (Parser Generator) (type "javacc" with no arguments for help) Warning: Bad option "-grammar_encoding=UTF-8" will be ignored. Reading from file <https://builds.apache.org/job/beam_PerformanceTests_TextIOIT/ws/runs/e6878d83/beam/sdks/java/extensions/sql/build/generated/fmpp/javacc/Parser.jj> . . . Note: UNICODE_INPUT option is specified. Please make sure you create the parser/lexer using a Reader with the correct character encoding. Warning: Lookahead adequacy checking not being performed since option LOOKAHEAD is more than 1. Set option FORCE_LA_CHECK to true to force checking. File "TokenMgrError.java" does not exist. Will create one. File "ParseException.java" does not exist. Will create one. File "Token.java" does not exist. Will create one. File "SimpleCharStream.java" does not exist. Will create one. Parser generated with 0 errors and 1 warnings. > Task :beam-model-job-management:extractIncludeProto > Task :beam-model-fn-execution:extractIncludeProto > Task :beam-model-pipeline:jar > Task :beam-model-job-management:generateProto > Task :beam-model-pipeline:extractIncludeTestProto > Task :beam-model-pipeline:extractTestProto > Task :beam-model-pipeline:generateTestProto NO-SOURCE > Task :beam-model-pipeline:compileTestJava NO-SOURCE > Task :beam-model-pipeline:processTestResources NO-SOURCE > Task :beam-model-pipeline:testClasses UP-TO-DATE > Task :beam-model-pipeline:packageTests > Task :beam-model-fn-execution:generateProto > Task :beam-model-pipeline:install > Task :beam-model-job-management:compileJava FROM-CACHE > Task :beam-model-job-management:classes > Task :beam-model-fn-execution:compileJava FROM-CACHE > Task :beam-model-fn-execution:classes > Task :beam-sdks-java-extensions-sql:processResources > Task :beam-sdks-java-extensions-sql:processTestResources NO-SOURCE > Task :beam-sdks-python:setupVirtualenv Using cached https://files.pythonhosted.org/packages/05/f6/0296e29b1bac6f85d2a8556d48adf825307f73109a3c2c17fb734292db0a/grpcio_tools-1.3.5-cp27-cp27mu-manylinux1_x86_64.whl Requirement not upgraded as not directly required: pluggy<1.0,>=0.3.0 in /home/jenkins/.local/lib/python2.7/site-packages (from tox==3.0.0) (0.6.0) Requirement not upgraded as not directly required: six in /usr/local/lib/python2.7/dist-packages (from tox==3.0.0) (1.11.0) Requirement not upgraded as not directly required: virtualenv>=1.11.2 in /home/jenkins/.local/lib/python2.7/site-packages (from tox==3.0.0) (15.2.0) Requirement not upgraded as not directly required: py>=1.4.17 in /home/jenkins/.local/lib/python2.7/site-packages (from tox==3.0.0) (1.5.3) Collecting grpcio>=1.3.5 (from grpcio-tools==1.3.5) Using cached https://files.pythonhosted.org/packages/0d/54/b647a6323be6526be27b2c90bb042769f1a7a6e59bd1a5f2eeb795bfece4/grpcio-1.11.0-cp27-cp27mu-manylinux1_x86_64.whl Collecting protobuf>=3.2.0 (from grpcio-tools==1.3.5) Using cached https://files.pythonhosted.org/packages/9d/61/54c3a9cfde6ffe0ca6a1786ddb8874263f4ca32e7693ad383bd8cf935015/protobuf-3.5.2.post1-cp27-cp27mu-manylinux1_x86_64.whl Requirement not upgraded as not directly required: enum34>=1.0.4 in /usr/local/lib/python2.7/dist-packages (from grpcio>=1.3.5->grpcio-tools==1.3.5) (1.1.6) Collecting futures>=2.2.0 (from grpcio>=1.3.5->grpcio-tools==1.3.5) Using cached https://files.pythonhosted.org/packages/2d/99/b2c4e9d5a30f6471e410a146232b4118e697fa3ffc06d6a65efde84debd0/futures-3.2.0-py2-none-any.whl Requirement not upgraded as not directly required: setuptools in /usr/local/lib/python2.7/dist-packages (from protobuf>=3.2.0->grpcio-tools==1.3.5) (39.0.1) Installing collected packages: protobuf, futures, grpcio, grpcio-tools Could not install packages due to an EnvironmentError: [Errno 13] Permission denied: '/usr/local/lib/python2.7/dist-packages/protobuf-3.5.2.post1-py2.7-nspkg.pth' Consider using the `--user` option or check the permissions. > Task :beam-sdks-python:setupVirtualenv FAILED > Task :beam-model-job-management:shadowJar > Task :beam-model-fn-execution:shadowJar > Task :beam-sdks-java-core:shadowJar Deprecated Gradle features were used in this build, making it incompatible with Gradle 5.0. See https://docs.gradle.org/4.7/userguide/command_line_interface.html#sec:command_line_warnings BUILD FAILED in 56s 132 actionable tasks: 125 executed, 5 from cache, 2 up-to-date Publishing build scan... https://gradle.com/s/sifyfojutjyd4 STDERR: FAILURE: Build failed with an exception. * Where: Build file '<https://builds.apache.org/job/beam_PerformanceTests_TextIOIT/ws/runs/e6878d83/beam/sdks/python/build.gradle'> line: 36 * What went wrong: Execution failed for task ':beam-sdks-python:setupVirtualenv'. > Process 'command 'sh'' finished with non-zero exit value 1 * Try: Run with --info or --debug option to get more log output. Run with --scan to get full insights. * Exception is: org.gradle.api.tasks.TaskExecutionException: Execution failed for task ':beam-sdks-python:setupVirtualenv'. at org.gradle.api.internal.tasks.execution.ExecuteActionsTaskExecuter.executeActions(ExecuteActionsTaskExecuter.java:103) at org.gradle.api.internal.tasks.execution.ExecuteActionsTaskExecuter.execute(ExecuteActionsTaskExecuter.java:73) at org.gradle.api.internal.tasks.execution.OutputDirectoryCreatingTaskExecuter.execute(OutputDirectoryCreatingTaskExecuter.java:51) at org.gradle.api.internal.tasks.execution.SkipCachedTaskExecuter.execute(SkipCachedTaskExecuter.java:105) at org.gradle.api.internal.tasks.execution.SkipUpToDateTaskExecuter.execute(SkipUpToDateTaskExecuter.java:59) at org.gradle.api.internal.tasks.execution.ResolveTaskOutputCachingStateExecuter.execute(ResolveTaskOutputCachingStateExecuter.java:54) at org.gradle.api.internal.tasks.execution.ResolveBuildCacheKeyExecuter.execute(ResolveBuildCacheKeyExecuter.java:66) at org.gradle.api.internal.tasks.execution.ValidatingTaskExecuter.execute(ValidatingTaskExecuter.java:59) at org.gradle.api.internal.tasks.execution.SkipEmptySourceFilesTaskExecuter.execute(SkipEmptySourceFilesTaskExecuter.java:101) at org.gradle.api.internal.tasks.execution.FinalizeInputFilePropertiesTaskExecuter.execute(FinalizeInputFilePropertiesTaskExecuter.java:44) at org.gradle.api.internal.tasks.execution.CleanupStaleOutputsExecuter.execute(CleanupStaleOutputsExecuter.java:91) at org.gradle.api.internal.tasks.execution.ResolveTaskArtifactStateTaskExecuter.execute(ResolveTaskArtifactStateTaskExecuter.java:62) at org.gradle.api.internal.tasks.execution.SkipTaskWithNoActionsExecuter.execute(SkipTaskWithNoActionsExecuter.java:59) at org.gradle.api.internal.tasks.execution.SkipOnlyIfTaskExecuter.execute(SkipOnlyIfTaskExecuter.java:54) at org.gradle.api.internal.tasks.execution.ExecuteAtMostOnceTaskExecuter.execute(ExecuteAtMostOnceTaskExecuter.java:43) at org.gradle.api.internal.tasks.execution.CatchExceptionTaskExecuter.execute(CatchExceptionTaskExecuter.java:34) at org.gradle.execution.taskgraph.DefaultTaskGraphExecuter$EventFiringTaskWorker$1.run(DefaultTaskGraphExecuter.java:256) at org.gradle.internal.operations.DefaultBuildOperationExecutor$RunnableBuildOperationWorker.execute(DefaultBuildOperationExecutor.java:317) at org.gradle.internal.operations.DefaultBuildOperationExecutor$RunnableBuildOperationWorker.execute(DefaultBuildOperationExecutor.java:309) at org.gradle.internal.operations.DefaultBuildOperationExecutor.execute(DefaultBuildOperationExecutor.java:185) at org.gradle.internal.operations.DefaultBuildOperationExecutor.run(DefaultBuildOperationExecutor.java:97) at org.gradle.internal.operations.DelegatingBuildOperationExecutor.run(DelegatingBuildOperationExecutor.java:31) at org.gradle.execution.taskgraph.DefaultTaskGraphExecuter$EventFiringTaskWorker.execute(DefaultTaskGraphExecuter.java:249) at org.gradle.execution.taskgraph.DefaultTaskGraphExecuter$EventFiringTaskWorker.execute(DefaultTaskGraphExecuter.java:238) at org.gradle.execution.taskgraph.DefaultTaskPlanExecutor$TaskExecutorWorker$1.execute(DefaultTaskPlanExecutor.java:104) at org.gradle.execution.taskgraph.DefaultTaskPlanExecutor$TaskExecutorWorker$1.execute(DefaultTaskPlanExecutor.java:98) at org.gradle.execution.taskgraph.DefaultTaskExecutionPlan.execute(DefaultTaskExecutionPlan.java:663) at org.gradle.execution.taskgraph.DefaultTaskExecutionPlan.executeWithTask(DefaultTaskExecutionPlan.java:596) at org.gradle.execution.taskgraph.DefaultTaskPlanExecutor$TaskExecutorWorker.run(DefaultTaskPlanExecutor.java:98) at org.gradle.internal.concurrent.ExecutorPolicy$CatchAndRecordFailures.onExecute(ExecutorPolicy.java:63) at org.gradle.internal.concurrent.ManagedExecutorImpl$1.run(ManagedExecutorImpl.java:46) at org.gradle.internal.concurrent.ThreadFactoryImpl$ManagedThreadRunnable.run(ThreadFactoryImpl.java:55) Caused by: org.gradle.process.internal.ExecException: Process 'command 'sh'' finished with non-zero exit value 1 at org.gradle.process.internal.DefaultExecHandle$ExecResultImpl.assertNormalExitValue(DefaultExecHandle.java:389) at org.gradle.process.internal.DefaultExecAction.execute(DefaultExecAction.java:36) at org.gradle.api.internal.file.DefaultFileOperations.exec(DefaultFileOperations.java:192) at org.gradle.api.internal.project.DefaultProject.exec(DefaultProject.java:1087) at org.gradle.groovy.scripts.DefaultScript.exec(DefaultScript.java:253) at org.gradle.internal.metaobject.BeanDynamicObject$MetaClassAdapter.invokeMethod(BeanDynamicObject.java:479) at org.gradle.internal.metaobject.BeanDynamicObject.tryInvokeMethod(BeanDynamicObject.java:191) at org.gradle.groovy.scripts.BasicScript$ScriptDynamicObject.tryInvokeMethod(BasicScript.java:130) at org.gradle.internal.metaobject.ConfigureDelegate.invokeMethod(ConfigureDelegate.java:78) at build_3xt3r79nau3a48pd14rq3e96u$_run_closure2$_closure16.doCall(<https://builds.apache.org/job/beam_PerformanceTests_TextIOIT/ws/runs/e6878d83/beam/sdks/python/build.gradle>:36) at org.gradle.api.internal.AbstractTask$ClosureTaskAction.execute(AbstractTask.java:732) at org.gradle.api.internal.AbstractTask$ClosureTaskAction.execute(AbstractTask.java:705) at org.gradle.api.internal.tasks.execution.ExecuteActionsTaskExecuter$1.run(ExecuteActionsTaskExecuter.java:124) at org.gradle.internal.operations.DefaultBuildOperationExecutor$RunnableBuildOperationWorker.execute(DefaultBuildOperationExecutor.java:317) at org.gradle.internal.operations.DefaultBuildOperationExecutor$RunnableBuildOperationWorker.execute(DefaultBuildOperationExecutor.java:309) at org.gradle.internal.operations.DefaultBuildOperationExecutor.execute(DefaultBuildOperationExecutor.java:185) at org.gradle.internal.operations.DefaultBuildOperationExecutor.run(DefaultBuildOperationExecutor.java:97) at org.gradle.internal.operations.DelegatingBuildOperationExecutor.run(DelegatingBuildOperationExecutor.java:31) at org.gradle.api.internal.tasks.execution.ExecuteActionsTaskExecuter.executeAction(ExecuteActionsTaskExecuter.java:113) at org.gradle.api.internal.tasks.execution.ExecuteActionsTaskExecuter.executeActions(ExecuteActionsTaskExecuter.java:95) ... 31 more * Get more help at https://help.gradle.org 2018-05-09 00:24:05,931 e6878d83 MainThread beam_integration_benchmark(1/1) INFO Running benchmark beam_integration_benchmark 2018-05-09 00:24:05,943 e6878d83 MainThread beam_integration_benchmark(1/1) INFO Running: <https://builds.apache.org/job/beam_PerformanceTests_TextIOIT/ws/runs/e6878d83/beam/gradlew> integrationTest --tests=org.apache.beam.sdk.io.text.TextIOIT -p sdks/java/io/file-based-io-tests -DintegrationTestRunner=dataflow -Dfilesystem=gcs -DintegrationTestPipelineOptions=["--project=apache-beam-testing","--tempRoot=gs://temp-storage-for-perf-tests","--filenamePrefix=gs://temp-storage-for-perf-tests/beam_PerformanceTests_TextIOIT/488/","--numberOfRecords=1000000","--runner=TestDataflowRunner"] --stacktrace --info 2018-05-09 00:31:26,900 e6878d83 MainThread beam_integration_benchmark(1/1) INFO Cleaning up benchmark beam_integration_benchmark 2018-05-09 00:31:26,901 e6878d83 MainThread beam_integration_benchmark(1/1) INFO Tearing down resources for benchmark beam_integration_benchmark 2018-05-09 00:31:26,916 e6878d83 MainThread INFO Publishing 2 samples to <https://builds.apache.org/job/beam_PerformanceTests_TextIOIT/ws/runs/e6878d83/perfkit-bq-pubRipVhq.json> 2018-05-09 00:31:26,916 e6878d83 MainThread INFO Publishing 2 samples to beam_performance.textioit_pkb_results 2018-05-09 00:31:26,916 e6878d83 MainThread INFO Running: bq load --autodetect --source_format=NEWLINE_DELIMITED_JSON beam_performance.textioit_pkb_results <https://builds.apache.org/job/beam_PerformanceTests_TextIOIT/ws/runs/e6878d83/perfkit-bq-pubRipVhq.json> 2018-05-09 00:31:35,906 e6878d83 MainThread INFO -------------------------PerfKitBenchmarker Complete Results------------------------- {'metadata': {'dpb_cluster_id': None, 'dpb_cluster_shape': 'n1-standard-1', 'dpb_cluster_size': 2, 'dpb_dataflow_runner': 'DataflowRunner', 'dpb_dataflow_sdk': None, 'dpb_service': 'dataflow', 'perfkitbenchmarker_version': 'v1.12.0-580-g36cfc82', 'run_number': 0}, 'metric': 'run_time', 'official': True, 'owner': 'jenkins', 'product_name': 'PerfKitBenchmarker', 'run_uri': 'e6878d83-db758041-a9b2-4ed9-93e4-286ceb334d6c', 'sample_uri': '9f091313-c5b4-4dc1-a959-389497d300f4', 'test': 'beam_integration_benchmark', 'timestamp': 1525825886.899967, 'unit': 'seconds', 'value': 440.967753} {'metadata': {'perfkitbenchmarker_version': 'v1.12.0-580-g36cfc82'}, 'metric': 'End to End Runtime', 'official': True, 'owner': 'jenkins', 'product_name': 'PerfKitBenchmarker', 'run_uri': 'e6878d83-db758041-a9b2-4ed9-93e4-286ceb334d6c', 'sample_uri': 'f77a6488-d9c4-4b9a-b25c-0a0452043664', 'test': 'beam_integration_benchmark', 'timestamp': 1525825886.90162, 'unit': 'seconds', 'value': 504.1220610141754} -------------------------PerfKitBenchmarker Results Summary------------------------- BEAM_INTEGRATION_BENCHMARK: dpb_cluster_shape="n1-standard-1" dpb_cluster_size="2" dpb_dataflow_runner="DataflowRunner" dpb_service="dataflow" run_number="0" run_time 440.967753 seconds End to End Runtime 504.122061 seconds ------------------------- For all tests: dpb_cluster_id="None" dpb_dataflow_sdk="None" perfkitbenchmarker_version="v1.12.0-580-g36cfc82" 2018-05-09 00:31:35,907 e6878d83 MainThread INFO Publishing 2 samples to <https://builds.apache.org/job/beam_PerformanceTests_TextIOIT/ws/runs/e6878d83/perfkitbenchmarker_results.json> 2018-05-09 00:31:35,907 e6878d83 MainThread INFO Benchmark run statuses: ------------------------------------------------------------------------------------ Name UID Status Failed Substatus ------------------------------------------------------------------------------------ beam_integration_benchmark beam_integration_benchmark0 SUCCEEDED ------------------------------------------------------------------------------------ Success rate: 100.00% (1/1) 2018-05-09 00:31:35,907 e6878d83 MainThread INFO Complete logs can be found at: <https://builds.apache.org/job/beam_PerformanceTests_TextIOIT/ws/runs/e6878d83/pkb.log> 2018-05-09 00:31:35,908 e6878d83 MainThread INFO Completion statuses can be found at: <https://builds.apache.org/job/beam_PerformanceTests_TextIOIT/ws/runs/e6878d83/completion_statuses.json> [Set GitHub commit status (universal)] SUCCESS on repos [GHRepository@a30257c[description=Apache Beam,homepage=,name=beam,fork=false,size=58753,milestones={},language=Java,commits={},source=<null>,parent=<null>,responseHeaderFields={null=[HTTP/1.1 200 OK], Access-Control-Allow-Origin=[*], Access-Control-Expose-Headers=[ETag, Link, Retry-After, X-GitHub-OTP, X-RateLimit-Limit, X-RateLimit-Remaining, X-RateLimit-Reset, X-OAuth-Scopes, X-Accepted-OAuth-Scopes, X-Poll-Interval], Cache-Control=[private, max-age=60, s-maxage=60], Content-Encoding=[gzip], Content-Security-Policy=[default-src 'none'], Content-Type=[application/json; charset=utf-8], Date=[Wed, 09 May 2018 00:31:38 GMT], ETag=[W/"f61e7a401b5666f651eab561fff376f2"], Last-Modified=[Wed, 09 May 2018 00:26:13 GMT], OkHttp-Received-Millis=[1525825901705], OkHttp-Response-Source=[NETWORK 200], OkHttp-Selected-Protocol=[http/1.1], OkHttp-Sent-Millis=[1525825898403], Referrer-Policy=[origin-when-cross-origin, strict-origin-when-cross-origin], Server=[GitHub.com], Status=[200 OK], Strict-Transport-Security=[max-age=31536000; includeSubdomains; preload], Transfer-Encoding=[chunked], Vary=[Accept, Authorization, Cookie, X-GitHub-OTP], X-Accepted-OAuth-Scopes=[repo], X-Content-Type-Options=[nosniff], X-Frame-Options=[deny], X-GitHub-Media-Type=[github.v3; format=json], X-GitHub-Request-Id=[AE3A:4F41:A6B67A:16D18B0:5AF2416A], X-OAuth-Scopes=[admin:repo_hook, repo, repo:status], X-RateLimit-Limit=[5000], X-RateLimit-Remaining=[4995], X-RateLimit-Reset=[1525828827], X-Runtime-rack=[0.065846], X-XSS-Protection=[1; mode=block]},url=https://api.github.com/repos/apache/beam,id=50904245]] (sha:7848764) with context:beam_PerformanceTests_TextIOIT Setting commit status on GitHub for https://github.com/apache/beam/commit/7848764a070b3bcd02528014838792b88c5ea2e4 ERROR: Build step failed with exception java.io.FileNotFoundException: https://api.github.com/repos/apache/beam/statuses/7848764a070b3bcd02528014838792b88c5ea2e4 at com.squareup.okhttp.internal.huc.HttpURLConnectionImpl.getInputStream(HttpURLConnectionImpl.java:243) at com.squareup.okhttp.internal.huc.DelegatingHttpsURLConnection.getInputStream(DelegatingHttpsURLConnection.java:210) at com.squareup.okhttp.internal.huc.HttpsURLConnectionImpl.getInputStream(HttpsURLConnectionImpl.java:25) at org.kohsuke.github.Requester.parse(Requester.java:612) at org.kohsuke.github.Requester.parse(Requester.java:594) at org.kohsuke.github.Requester._to(Requester.java:272) Caused: org.kohsuke.github.GHFileNotFoundException: {"message":"Not Found","documentation_url":"https://developer.github.com/v3/repos/statuses/#create-a-status"} at org.kohsuke.github.Requester.handleApiError(Requester.java:686) at org.kohsuke.github.Requester._to(Requester.java:293) at org.kohsuke.github.Requester.to(Requester.java:234) at org.kohsuke.github.GHRepository.createCommitStatus(GHRepository.java:1075) at org.jenkinsci.plugins.github.status.GitHubCommitStatusSetter.perform(GitHubCommitStatusSetter.java:160) Caused: org.jenkinsci.plugins.github.common.CombineErrorHandler$ErrorHandlingException at org.jenkinsci.plugins.github.common.CombineErrorHandler.handle(CombineErrorHandler.java:74) at org.jenkinsci.plugins.github.status.GitHubCommitStatusSetter.perform(GitHubCommitStatusSetter.java:164) at com.cloudbees.jenkins.GitHubCommitNotifier.perform(GitHubCommitNotifier.java:151) at hudson.tasks.BuildStepCompatibilityLayer.perform(BuildStepCompatibilityLayer.java:81) at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20) at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:744) at hudson.model.AbstractBuild$AbstractBuildExecution.performAllBuildSteps(AbstractBuild.java:690) at hudson.model.Build$BuildExecution.post2(Build.java:186) at hudson.model.AbstractBuild$AbstractBuildExecution.post(AbstractBuild.java:635) at hudson.model.Run.execute(Run.java:1749) at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43) at hudson.model.ResourceController.execute(ResourceController.java:97) at hudson.model.Executor.run(Executor.java:429) Build step 'Set build status on GitHub commit [deprecated]' marked build as failure
