See <https://ci-beam.apache.org/job/beam_LoadTests_Java_ParDo_Dataflow_Streaming/466/display/redirect?page=changes>
Changes: [Udi Meiri] Color Jenkins logs support [ningk] Added a whitespace lint precommit job. [ningk] Removed trailing whitespaces from all markdown and gradle build files. [ningk] Updated jenkins precommit job README, added comments of what needs to be [ningk] Removed trailing whitespaces from HEAD changes [Alan Myrvold] [BEAM-10049] Add licenses for go dependencies in python, java, and go [Robin Qiu] Fix bug in JoinScanWithRefConverter [noreply] Propagate BigQuery streaming insert throttled time to Dataflow worker in [tysonjh] [BEAM-10751] Revert "Extending archiveJunit post-commit task with [noreply] [BEAM-10764] Make is_in_ipython robust (#12641) [noreply] [BEAM-10766] Fix flake where ByteString coercion could modify cached [ningk] [BEAM-10771] Added Whitespacelint job to pull request template [noreply] Use setCoder, not setSchema (#12662) [noreply] [BEAM-10703] Prepare Dataflow Java runner for shardable states (#12578) [noreply] [BEAM-10549] Improve runtime type checking performance for the Python [Kyle Weaver] [BEAM-10460] Remove SparkPortableExecutionTest altogether. [noreply] [BEAM-10788] Also disable immutability checking for transforms within [noreply] [BEAM-5715, BEAM-8862, BEAM-8702] Removes all references to grpc_all and [noreply] [BEAM-10761] Prefer TestPubsub#assertSubscriptionEventuallyExists when [noreply] [BEAM-10654] Implemented ExternalSchemaIOTransformRegistrar for jdbc [noreply] Merge pull request #12580 from [BEAM-2855] nexmark python suite [noreply] [BEAM-9918] Adding tests and documentation to xlang components (#12667) ------------------------------------------ [...truncated 114.17 KB...] INFO: Adding Step: 108 as step s113 Aug 22, 2020 12:12:53 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding Step: 109 as step s114 Aug 22, 2020 12:12:53 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding Step: 110 as step s115 Aug 22, 2020 12:12:53 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding Step: 111 as step s116 Aug 22, 2020 12:12:53 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding Step: 112 as step s117 Aug 22, 2020 12:12:53 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding Step: 113 as step s118 Aug 22, 2020 12:12:53 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding Step: 114 as step s119 Aug 22, 2020 12:12:53 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding Step: 115 as step s120 Aug 22, 2020 12:12:53 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding Step: 116 as step s121 Aug 22, 2020 12:12:53 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding Step: 117 as step s122 Aug 22, 2020 12:12:53 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding Step: 118 as step s123 Aug 22, 2020 12:12:53 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding Step: 119 as step s124 Aug 22, 2020 12:12:53 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding Step: 120 as step s125 Aug 22, 2020 12:12:53 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding Step: 121 as step s126 Aug 22, 2020 12:12:53 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding Step: 122 as step s127 Aug 22, 2020 12:12:53 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding Step: 123 as step s128 Aug 22, 2020 12:12:53 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding Step: 124 as step s129 Aug 22, 2020 12:12:53 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding Step: 125 as step s130 Aug 22, 2020 12:12:53 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding Step: 126 as step s131 Aug 22, 2020 12:12:53 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding Step: 127 as step s132 Aug 22, 2020 12:12:53 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding Step: 128 as step s133 Aug 22, 2020 12:12:53 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding Step: 129 as step s134 Aug 22, 2020 12:12:53 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding Step: 130 as step s135 Aug 22, 2020 12:12:53 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding Step: 131 as step s136 Aug 22, 2020 12:12:53 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding Step: 132 as step s137 Aug 22, 2020 12:12:53 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding Step: 133 as step s138 Aug 22, 2020 12:12:53 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding Step: 134 as step s139 Aug 22, 2020 12:12:53 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding Step: 135 as step s140 Aug 22, 2020 12:12:53 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding Step: 136 as step s141 Aug 22, 2020 12:12:53 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding Step: 137 as step s142 Aug 22, 2020 12:12:53 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding Step: 138 as step s143 Aug 22, 2020 12:12:53 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding Step: 139 as step s144 Aug 22, 2020 12:12:53 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding Step: 140 as step s145 Aug 22, 2020 12:12:53 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding Step: 141 as step s146 Aug 22, 2020 12:12:53 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding Step: 142 as step s147 Aug 22, 2020 12:12:53 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding Step: 143 as step s148 Aug 22, 2020 12:12:53 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding Step: 144 as step s149 Aug 22, 2020 12:12:53 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding Step: 145 as step s150 Aug 22, 2020 12:12:53 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding Step: 146 as step s151 Aug 22, 2020 12:12:53 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding Step: 147 as step s152 Aug 22, 2020 12:12:53 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding Step: 148 as step s153 Aug 22, 2020 12:12:53 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding Step: 149 as step s154 Aug 22, 2020 12:12:53 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding Step: 150 as step s155 Aug 22, 2020 12:12:53 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding Step: 151 as step s156 Aug 22, 2020 12:12:53 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding Step: 152 as step s157 Aug 22, 2020 12:12:53 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding Step: 153 as step s158 Aug 22, 2020 12:12:53 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding Step: 154 as step s159 Aug 22, 2020 12:12:53 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding Step: 155 as step s160 Aug 22, 2020 12:12:53 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding Step: 156 as step s161 Aug 22, 2020 12:12:53 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding Step: 157 as step s162 Aug 22, 2020 12:12:53 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding Step: 158 as step s163 Aug 22, 2020 12:12:53 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding Step: 159 as step s164 Aug 22, 2020 12:12:53 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding Step: 160 as step s165 Aug 22, 2020 12:12:53 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding Step: 161 as step s166 Aug 22, 2020 12:12:53 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding Step: 162 as step s167 Aug 22, 2020 12:12:53 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding Step: 163 as step s168 Aug 22, 2020 12:12:53 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding Step: 164 as step s169 Aug 22, 2020 12:12:53 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding Step: 165 as step s170 Aug 22, 2020 12:12:53 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding Step: 166 as step s171 Aug 22, 2020 12:12:53 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding Step: 167 as step s172 Aug 22, 2020 12:12:53 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding Step: 168 as step s173 Aug 22, 2020 12:12:53 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding Step: 169 as step s174 Aug 22, 2020 12:12:53 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding Step: 170 as step s175 Aug 22, 2020 12:12:53 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding Step: 171 as step s176 Aug 22, 2020 12:12:53 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding Step: 172 as step s177 Aug 22, 2020 12:12:53 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding Step: 173 as step s178 Aug 22, 2020 12:12:53 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding Step: 174 as step s179 Aug 22, 2020 12:12:53 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding Step: 175 as step s180 Aug 22, 2020 12:12:53 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding Step: 176 as step s181 Aug 22, 2020 12:12:53 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding Step: 177 as step s182 Aug 22, 2020 12:12:53 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding Step: 178 as step s183 Aug 22, 2020 12:12:53 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding Step: 179 as step s184 Aug 22, 2020 12:12:53 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding Step: 180 as step s185 Aug 22, 2020 12:12:53 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding Step: 181 as step s186 Aug 22, 2020 12:12:53 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding Step: 182 as step s187 Aug 22, 2020 12:12:53 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding Step: 183 as step s188 Aug 22, 2020 12:12:53 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding Step: 184 as step s189 Aug 22, 2020 12:12:53 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding Step: 185 as step s190 Aug 22, 2020 12:12:53 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding Step: 186 as step s191 Aug 22, 2020 12:12:53 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding Step: 187 as step s192 Aug 22, 2020 12:12:53 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding Step: 188 as step s193 Aug 22, 2020 12:12:53 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding Step: 189 as step s194 Aug 22, 2020 12:12:53 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding Step: 190 as step s195 Aug 22, 2020 12:12:53 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding Step: 191 as step s196 Aug 22, 2020 12:12:53 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding Step: 192 as step s197 Aug 22, 2020 12:12:53 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding Step: 193 as step s198 Aug 22, 2020 12:12:53 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding Step: 194 as step s199 Aug 22, 2020 12:12:53 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding Step: 195 as step s200 Aug 22, 2020 12:12:53 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding Step: 196 as step s201 Aug 22, 2020 12:12:53 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding Step: 197 as step s202 Aug 22, 2020 12:12:53 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding Step: 198 as step s203 Aug 22, 2020 12:12:53 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding Step: 199 as step s204 Aug 22, 2020 12:12:53 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding ParDo(TimeMonitor)2 as step s205 Aug 22, 2020 12:12:53 PM org.apache.beam.runners.dataflow.DataflowRunner run INFO: Staging pipeline description to gs://temp-storage-for-perf-tests/loadtests/staging/ Aug 22, 2020 12:12:53 PM org.apache.beam.runners.dataflow.util.PackageUtil tryStagePackage INFO: Uploading <373155 bytes, hash 56274292fb2ddd0dd37f34e76874c80eeed137fac78686b9c35d03f51c582b91> to gs://temp-storage-for-perf-tests/loadtests/staging/pipeline-VidCkvst3Q3TfzTnaHTIDu7RN_rHhoa5w10D9RxYK5E.pb Aug 22, 2020 12:12:54 PM org.apache.beam.runners.dataflow.DataflowRunner run INFO: Dataflow SDK version: 2.25.0-SNAPSHOT Aug 22, 2020 12:12:55 PM org.apache.beam.runners.dataflow.DataflowRunner run INFO: To access the Dataflow monitoring console, please navigate to https://console.cloud.google.com/dataflow/jobs/us-central1/2020-08-22_05_12_54-6102283971490627755?project=apache-beam-testing Aug 22, 2020 12:12:55 PM org.apache.beam.runners.dataflow.DataflowRunner run INFO: Submitted job: 2020-08-22_05_12_54-6102283971490627755 Aug 22, 2020 12:12:55 PM org.apache.beam.runners.dataflow.DataflowRunner run INFO: To cancel the job using the 'gcloud' tool, run: > gcloud dataflow jobs --project=apache-beam-testing cancel > --region=us-central1 2020-08-22_05_12_54-6102283971490627755 Aug 22, 2020 12:12:58 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process WARNING: 2020-08-22T12:12:57.762Z: The workflow name is not a valid Cloud Label. Labels applied to Cloud resources (such as GCE Instances) for monitoring will be labeled with this modified job name: load0tests0java0dataflow0streaming0pardo02-jenkins-0822121-x98b. For the best monitoring experience, please name your job with a valid Cloud Label. For details, see: https://cloud.google.com/compute/docs/labeling-resources#restrictions Aug 22, 2020 12:13:05 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process SEVERE: 2020-08-22T12:13:04.178Z: Workflow failed. Causes: The Compute Engine API has not fully initialized. Please try again in a few minutes., Unable to get machine type information for machine type n1-standard-4 in zone us-central1-f because of insufficient permissions. Please refer to https://cloud.google.com/dataflow/access-control#creating_jobs and make sure you have sufficient permissions. Aug 22, 2020 12:13:05 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2020-08-22T12:13:04.193Z: Cleaning up. Aug 22, 2020 12:13:05 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2020-08-22T12:13:04.235Z: Worker pool stopped. Aug 22, 2020 12:13:05 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2020-08-22T12:13:04.251Z: Stopping **** pool... Aug 22, 2020 12:13:07 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process WARNING: 2020-08-22T12:13:07.201Z: Your project already contains 100 Dataflow-created metric descriptors and Stackdriver will not create new Dataflow custom metrics for this job. Each unique user-defined metric name (independent of the DoFn in which it is defined) produces a new metric descriptor. To delete old / unused metric descriptors see https://developers.google.com/apis-explorer/#p/monitoring/v3/monitoring.projects.metricDescriptors.list and https://developers.google.com/apis-explorer/#p/monitoring/v3/monitoring.projects.metricDescriptors.delete Aug 22, 2020 12:13:11 PM org.apache.beam.runners.dataflow.DataflowPipelineJob logTerminalState INFO: Job 2020-08-22_05_12_54-6102283971490627755 failed with status FAILED. Aug 22, 2020 12:13:11 PM org.apache.beam.sdk.testutils.metrics.MetricsReader getCounterMetric SEVERE: Failed to get metric totalBytes.count, from namespace pardo Load test results for test (ID): 0fa7a758-f108-44fb-acde-e2e22c57be17 and timestamp: 2020-08-22T12:12:50.323000000Z: Metric: Value: dataflow_runtime_sec 0.0 dataflow_total_bytes_count -1.0 Exception in thread "main" java.lang.RuntimeException: Invalid job state: FAILED. at org.apache.beam.sdk.loadtests.JobFailure.handleFailure(JobFailure.java:55) at org.apache.beam.sdk.loadtests.LoadTest.run(LoadTest.java:134) at org.apache.beam.sdk.loadtests.ParDoLoadTest.run(ParDoLoadTest.java:53) at org.apache.beam.sdk.loadtests.ParDoLoadTest.main(ParDoLoadTest.java:103) > Task :sdks:java:testing:load-tests:run FAILED FAILURE: Build failed with an exception. * What went wrong: Execution failed for task ':sdks:java:testing:load-tests:run'. > Process 'command '/usr/lib/jvm/java-8-openjdk-amd64/bin/java'' finished with > non-zero exit value 1 * Try: Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. Run with --scan to get full insights. * Get more help at https://help.gradle.org Deprecated Gradle features were used in this build, making it incompatible with Gradle 6.0. Use '--warning-mode all' to show the individual deprecation warnings. See https://docs.gradle.org/5.2.1/userguide/command_line_interface.html#sec:command_line_warnings BUILD FAILED in 27s 73 actionable tasks: 1 executed, 72 up-to-date Publishing build scan... https://gradle.com/s/74ebubc3fsuy6 Build step 'Invoke Gradle script' changed build result to FAILURE Build step 'Invoke Gradle script' marked build as failure --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
