See <https://ci-beam.apache.org/job/beam_LoadTests_Java_CoGBK_Dataflow_V2_Streaming_Java17/138/display/redirect?page=changes>
Changes: [kevinsijo] Setting up a basic directory [kevinsijo] Mirroring Python SDK's directory structure [kerrydc] Adds initial tests [kevinsijo] 'runners' is the correct directory name [Pablo Estrada] sketching the core API for JS SDK [jonathanlui] add .gitignore for node/ts project [Robert Bradshaw] Worker directory. [Robert Bradshaw] Fix complile errors with explicit any for callables. [Robert Bradshaw] Add worker entry point. [Robert Bradshaw] Add proto generation code. [Robert Bradshaw] Add generated proto files. [Robert Bradshaw] Attempts to get ts protos to compile. [Robert Bradshaw] Exclude ts protos for now. [Robert Bradshaw] More changes to get ts protos working. [Robert Bradshaw] Update scripts and config to get protos compiling. [Robert Bradshaw] Update geenrated files. [jonathanlui] add build and clean script to compile ts [Robert Bradshaw] Generate server for loopback worker. [Robert Bradshaw] Generated grpc servers for loopback. [Robert Bradshaw] Add typescript formatter. [Robert Bradshaw] Loopback server (that does nothing). [Robert Bradshaw] Working server. [Pablo Estrada] Starting expansion of primitive transforms [Pablo Estrada] Starting to implement and support standard coders [Robert Bradshaw] Also generate grpc clients. [Robert Bradshaw] Basic implementation of worker harness. [Pablo Estrada] fix the build [Robert Bradshaw] Add some missing files for worker harness. [Robert Bradshaw] Refactor operators to use registration. [jonathanlui] enable ts in mocha [jonathanlui] update readme [jonathanlui] --save-dev @types/mocha [jonathanlui] translate core_test.js to typescript [Robert Bradshaw] Encapsulate worker service in a class. [Kenneth Knowles] Port standard_coders_test to typescript (superficially) [Pablo Estrada] Starting the proto translation of Impulse, ParDo, GBK [Robert Bradshaw] Add some tests for the worker code. [Robert Bradshaw] Fixing old lock file error. [Pablo Estrada] Adding transform names and fixing GBK coder issue [Robert Bradshaw] npx tsfmt -r src/apache_beam/base.ts src/apache_beam/transforms/core.ts [Kenneth Knowles] switch to import style require() statements [Kenneth Knowles] Add Coder interface using protobufjs classes [Kenneth Knowles] BytesCoder with some failures [noreply] Added GeneralObjectCoder and using it as coder for most transforms (#9) [Kenneth Knowles] Fix order of arguments to deepEqual [Kenneth Knowles] Encode expected encoding as binary [Robert Bradshaw] Refactor API to allow for composites. [jrmccluskey] Initial setup for automated Java expansion startup [jrmccluskey] Update exp_service.ts [Kenneth Knowles] Fix up coder deserialization [Robert Bradshaw] Simplify GBK coder computation. [Robert Bradshaw] Remove top-level PValue. [Pablo Estrada] Make tests green [Robert Bradshaw] Rename PValueish to PValue. [jonathanlui] node runner [jonathanlui] whitespaces [Robert Bradshaw] Make Runner.run async. [jonathanlui] bson and fast-deep-equal should not be listed as devdependency [jrmccluskey] Add basic Dockerfile that starts ExternalWorkerPool [Robert Bradshaw] Direct runner. [kevinsijo] Testing expansion service communication [Robert Bradshaw] Added flatten, assertion checkers. [Pablo Estrada] progress on basic coders [Robert Bradshaw] Fixing the build. [Robert Bradshaw] Cleanup, simplify access. [Pablo Estrada] Adding limited support for KVCoder and IterableCoder [Robert Bradshaw] Introduce PipelineContext. [Robert Bradshaw] Add toProto to all coders. [Robert Bradshaw] Some work with coders. [Robert Bradshaw] Remove debug logging. [Robert Bradshaw] Use coders over data channel. [Kenneth Knowles] explicitly sequence sub-coder serializations [Kenneth Knowles] no more need to extend FakeCoder [Kenneth Knowles] actually advance reader [Kenneth Knowles] autoformat [Kenneth Knowles] protobufjs already can write and read signed varints [Kenneth Knowles] with improved test harness, kv has many more failures [Kenneth Knowles] read bytescoder from correct position [Kenneth Knowles] no more fake coders [Kenneth Knowles] varint examples all work [Kenneth Knowles] simplify coder value parsing [Kenneth Knowles] global window coder [Kenneth Knowles] fix swapEndian32 [Robert Bradshaw] Add P(...) operator. [kevinsijo] Implementing RowCoder encoding. [jrmccluskey] remove unused container dir [kevinsijo] Corrected sorting of encoded positions to reflect an argsort instead. [Robert Bradshaw] Populate environments. [kevinsijo] Implementing RowCoder decoding. [Kenneth Knowles] preliminary unbounded iterable coder [Kenneth Knowles] friendlier description of standard coder test case [Kenneth Knowles] fix test harness; iterable works [jrmccluskey] first pass at boot.go [jonathanlui] update package-lock.json [jonathanlui] make NodeRunner a subclass of Runner [jonathanlui] add waitUntilFinish interface member [Pablo Estrada] Adding double coder [Kenneth Knowles] scaffolding for windowed values [Pablo Estrada] Adding type information to PColleciton and PTransform [jonathanlui] fix direct runner [Pablo Estrada] Adding typing information for DoFns [Kenneth Knowles] add interval window [Robert Bradshaw] Export PValue. [Robert Bradshaw] Add CombineFn interface. [Robert Bradshaw] Typed flatten. [jonathanlui] add runAsync method to base.Runner [Kenneth Knowles] add Long package [Pablo Estrada] Adding more types. Making PValue typed [Kenneth Knowles] instant coder draft [Robert Bradshaw] Return job state from direct runner. [Kenneth Knowles] type instant = long [jonathanlui] implement NodeRunner.runPipeline [Kenneth Knowles] autoformat [kevinsijo] Completed implementation of basic row coder [Kenneth Knowles] Fix IntervalWindowCoder, almost [Kenneth Knowles] fix interval window coder [Kenneth Knowles] autoformat [Robert Bradshaw] loopback runner works [Kenneth Knowles] move core element types into values.ts [Kenneth Knowles] just build object directly to be cool [Robert Bradshaw] GBK working on ULR. [Robert Bradshaw] Async transforms. [Robert Bradshaw] External transform grpah splicing. [Kenneth Knowles] progress on windowed value: paneinfo encoding [Robert Bradshaw] Fix merge. [Robert Bradshaw] autoformat [Kenneth Knowles] full windowed value coder [kerrydc] Updates tests to use correct types, adds generics where needed to DoFns [Robert Bradshaw] Add serialization librarires.' [Robert Bradshaw] Add Split() PTransform, for producing multiple outputs from a single [Robert Bradshaw] Schema-encoded external payloads. [kevinsijo] Adding Schema inference from JSON [Pablo Estrada] Removing unused directories [Pablo Estrada] Support for finishBundle and improving typing annotations. [Pablo Estrada] A base implementation of combiners with GBK/ParDo [Robert Bradshaw] Fully propagate windowing information in both remote and direct runner. [Robert Bradshaw] Make args and kwargs optional for python external transform. [Robert Bradshaw] Infer schema for external transforms. [Pablo Estrada] Implementing a custom combine fn as an example. Small fixes [Robert Bradshaw] Fix missing windowing information in combiners. [Robert Bradshaw] PostShuffle needn't group by key as that's already done. [Robert Bradshaw] Guard pre-combine for global window only. [Robert Bradshaw] WindowInto [Robert Bradshaw] Fix optional kwargs. [Robert Bradshaw] A couple of tweaks for js + py [Robert Bradshaw] Add windowing file. [Robert Bradshaw] CombineBy transform, stand-alone WordCount. [Robert Bradshaw] cleanup [Robert Bradshaw] Actually fix optional external kwargs. [Robert Bradshaw] Demo2, textio read. [Robert Bradshaw] Add command lines for starting up the servers. [Robert Bradshaw] Run prettier on the full codebase. [Robert Bradshaw] Update deps. [Pablo Estrada] Adding docstrings for core.ts. Prettier dependency [Pablo Estrada] Documenting coder interfaces [Pablo Estrada] Added documentation for a few standard coders [Robert Bradshaw] Unified grouping and combining. [Robert Bradshaw] Allow PCollection ids to be lazy. [Robert Bradshaw] Reorganize module structure. [Robert Bradshaw] A couple more renames. [Robert Bradshaw] Simplify. [Robert Bradshaw] Consolidation. [Robert Bradshaw] Fix build. [Robert Bradshaw] Add optional context to ParDo. [Robert Bradshaw] fixup: iterable coder endian sign issue [Robert Bradshaw] omit context for map(console.log) [Robert Bradshaw] Fix ReadFromText coders. [Robert Bradshaw] Flesh out README with overview and current state. [noreply] Readme typo [Robert Bradshaw] Two more TODOs. [noreply] Add a pointer to the example wordcount to the readme. [Pablo Estrada] Documenting coders and implementing unknown-length method [Robert Bradshaw] UIID dependency. [Robert Bradshaw] Artifact handling. [Robert Bradshaw] Properly wait on data channel for bundle completion. [Robert Bradshaw] Automatic java expansion service startup. [Robert Bradshaw] Process promises. [Robert Bradshaw] Implement side inputs. [Robert Bradshaw] Cleanup. [Robert Bradshaw] Put complex constext stuff in its own file. [Robert Bradshaw] Rename BoundedWindow to just Window. [Robert Bradshaw] Alternative splitter class. [Pablo Estrada] Documenting internal functions [Robert Bradshaw] Take a pass clarifying the TODOs. [Robert Bradshaw] Sql transform wrapper. [Robert Bradshaw] Incorporate some feedback into the TODOs. [Robert Bradshaw] More TODOs. [Robert Bradshaw] Remove app placeholder. [Robert Bradshaw] Apache license headers. [Robert Bradshaw] More TODOs [jankuehle] Suggestions for TypeScript todos [dannymccormick] Add actions for typescript sdk [dannymccormick] Fix test command [noreply] Add missing version [dannymccormick] Fix codecovTest command [noreply] Only do prettier check on linux [noreply] Only get codecov on linux [Robert Bradshaw] Resolve some comments. [Robert Bradshaw] Fix compile errors. [Robert Bradshaw] Prettier. [Robert Bradshaw] Re-order expandInternal arguments pending unification. [Robert Bradshaw] More consistent and stricter PTransform naming. [Robert Bradshaw] Notes on explicit, if less idiomatic, use of classes. [Robert Bradshaw] Let DoFn be an interface rather than a class. [Robert Bradshaw] Provide DoFn context to start and finish bundle. [Robert Bradshaw] Optional promise code simplification. [Robert Bradshaw] Cleanup todos. [Robert Bradshaw] Avoid any type where not needed. [Robert Bradshaw] Apache RAT excludes for typescript. [Robert Bradshaw] Remove empty READMEs. [Robert Bradshaw] Add licences statement to readme files. [Robert Bradshaw] More RAT fixes. [Robert Bradshaw] Another unsupported coder. [Robert Bradshaw] Remove debugging code. [noreply] Fix automatic naming with code coverage. [Robert Bradshaw] Coders cleanup. [Robert Bradshaw] Add tests for RowCoder. [Robert Bradshaw] Normalize capitalization, comments. [Robert Bradshaw] Install typescript closure packages. [Robert Bradshaw] npm audit fix [Robert Bradshaw] Move more imports out of base. [Robert Bradshaw] Changes needed to compile with ts closure plugin. [Robert Bradshaw] Use ttsc and ts-closure-transform plugin. [Robert Bradshaw] Serialization registration to actually get serialization working. [Robert Bradshaw] Container images working on local runner. [Robert Bradshaw] Add a portable job server that proxies the Dataflow backend. (#17189) [Robert Bradshaw] Improvements to dataflow job service for non-Python jobs. [Robert Bradshaw] Get dataflow working. [Robert Bradshaw] User friendly pipeline options. [Robert Bradshaw] Less classes, more functions. [Robert Bradshaw] Add new nullable standard coder. [Robert Bradshaw] Make Apache Rat happy. [Robert Bradshaw] Disable broken codecov. [Robert Bradshaw] Remove last uses of base.ts. [Robert Bradshaw] Remove unneedd file. [Robert Bradshaw] Remove more uneeded/unused files. [Robert Bradshaw] Cleanup tests. [Robert Bradshaw] Minor cleanups to coder tests. [noreply] Quote pip install package name [noreply] [BEAM-14374] Fix module import error in FullyQualifiedNamedTransform [Robert Bradshaw] Addressing issues from the review. [noreply] Apply suggestions from code review. [Robert Bradshaw] Post-merge fixes. [dannymccormick] Delete tags.go [Robert Bradshaw] Update tests to use our actual serialization libraries. [Robert Bradshaw] Another pass at TODOs, removing finished items. [Heejong Lee] [BEAM-14146] Python Streaming job failing to drain with BigQueryIO write [Kenneth Knowles] Add parameter for service account impersonation in GCP credentials [Heejong Lee] add test [noreply] Merge pull request #17490 from [BEAM-14370] [Website] Add new page about [noreply] [BEAM-14332] Refactored cluster management for Flink on Dataproc [noreply] [BEAM-13988] Update mtime to use time.UnixMilli() calls (#17578) [noreply] Fixing patching error on missing dependencies (#17564) [noreply] Merge pull request #17517 from [BEAM-14383] Improve "FailedRows" errors [Heejong Lee] add test without mock ------------------------------------------ [...truncated 50.18 KB...] 2d05db1e2a4b: Preparing 3bc383470c05: Preparing e93827457889: Preparing 08fa02ce37eb: Preparing a037458de4e0: Preparing 662540b7db15: Waiting 1b1b55724774: Waiting bafdbe68e4ae: Preparing 0e2e00df1023: Waiting d507830c9b22: Waiting a13c519c6361: Preparing 08fa02ce37eb: Waiting 82227587df8f: Waiting e93827457889: Waiting 3bc383470c05: Waiting bafdbe68e4ae: Waiting a13c519c6361: Waiting 1252636fdc5b: Waiting b0bdfb1e5c67: Waiting 8db9d37c9268: Waiting 2d05db1e2a4b: Waiting a037458de4e0: Waiting 413656da2079: Pushed 98faf80f3452: Pushed 94899fdeb521: Pushed 3c9402bf3f9b: Pushed 1e44127ca56a: Pushed 662540b7db15: Pushed 104aa3e0eab7: Pushed 82227587df8f: Pushed d507830c9b22: Pushed 1252636fdc5b: Pushed 1b1b55724774: Pushed 3bc383470c05: Layer already exists e93827457889: Layer already exists b0bdfb1e5c67: Pushed 08fa02ce37eb: Layer already exists a037458de4e0: Layer already exists bafdbe68e4ae: Layer already exists a13c519c6361: Layer already exists 8db9d37c9268: Pushed 2d05db1e2a4b: Pushed 0e2e00df1023: Pushed 20220507123039: digest: sha256:a201febb4501343aed27f34781031d37aebfe84ff97646ce1d23d15df7f7d15a size: 4729 > Task :sdks:java:testing:load-tests:run May 07, 2022 12:32:44 PM org.apache.beam.runners.dataflow.options.DataflowPipelineOptions$StagingLocationFactory create INFO: No stagingLocation provided, falling back to gcpTempLocation May 07, 2022 12:32:44 PM org.apache.beam.runners.dataflow.DataflowRunner fromOptions INFO: PipelineOptions.filesToStage was not specified. Defaulting to files from the classpath: will stage 222 files. Enable logging at DEBUG level to see which files will be staged. May 07, 2022 12:32:45 PM org.apache.beam.sdk.Pipeline validate WARNING: The following transforms do not have stable unique names: Window.Into() May 07, 2022 12:32:45 PM org.apache.beam.runners.dataflow.DataflowRunner run INFO: Executing pipeline on the Dataflow Service, which will have billing implications related to Google Compute Engine usage and other Google Cloud Services. May 07, 2022 12:32:48 PM org.apache.beam.runners.dataflow.util.PackageUtil stageClasspathElements INFO: Uploading 222 files from PipelineOptions.filesToStage to staging location to prepare for execution. May 07, 2022 12:32:48 PM org.apache.beam.runners.dataflow.util.PackageUtil stageClasspathElements INFO: Staging files complete: 222 files cached, 0 files newly uploaded in 0 seconds May 07, 2022 12:32:48 PM org.apache.beam.runners.dataflow.DataflowRunner run INFO: Staging portable pipeline proto to gs://temp-storage-for-perf-tests/loadtests/staging/ May 07, 2022 12:32:48 PM org.apache.beam.runners.dataflow.util.PackageUtil tryStagePackage INFO: Uploading <120277 bytes, hash a6ab5cbe90343949e56d702d9a136db2fa719819f168bdcc66299093c41072f1> to gs://temp-storage-for-perf-tests/loadtests/staging/pipeline-pqtcvpA0OUnlbXAtmhNtsvpxmBnxaL3MZimQk8QQcvE.pb May 07, 2022 12:32:50 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding Read input/DataflowRunner.StreamingUnboundedRead.ReadWithIds as step s1 May 07, 2022 12:32:50 PM org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource split INFO: Split into 20 bundles of sizes: [org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@751ae8a4, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@235d659c, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@4232b34a, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@2da16263, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@f5ce0bb, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@47e51549, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@101a461c, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@360e9c06, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@5ebffb44, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@311ff287, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@7377781e, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@31db34da, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@109f8c7e, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@5edacf20, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@16a5eb6d, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@7e307087, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@1220ef43, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@1a8b81e8, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@234cff57, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@5e8507f1] May 07, 2022 12:32:50 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding Read input/StripIds as step s2 May 07, 2022 12:32:50 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding Collect start time metrics (input) as step s3 May 07, 2022 12:32:50 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding Window.Into()/Window.Assign as step s4 May 07, 2022 12:32:50 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding Read co-input/DataflowRunner.StreamingUnboundedRead.ReadWithIds as step s5 May 07, 2022 12:32:50 PM org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource split INFO: Split into 20 bundles of sizes: [org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@3bf54172, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@2c9a6717, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@7b3cde6f, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@6d091cad, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@7c663eaf, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@3bb5ceb, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@1e692555, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@3ba0ae41, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@76fe6cdc, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@2ffb3aec, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@786ff1cb, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@46039a21, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@431e86b1, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@35c4e864, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@32a2a6be, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@682af059, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@5f36c8e3, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@4da39ca9, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@6a9344f5, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@5584d9c6] May 07, 2022 12:32:50 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding Read co-input/StripIds as step s6 May 07, 2022 12:32:50 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding Collect start time metrics (co-input) as step s7 May 07, 2022 12:32:50 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding Window.Into()2/Window.Assign as step s8 May 07, 2022 12:32:50 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding CoGroupByKey/MakeUnionTable0 as step s9 May 07, 2022 12:32:50 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding CoGroupByKey/MakeUnionTable1 as step s10 May 07, 2022 12:32:50 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding CoGroupByKey/Flatten as step s11 May 07, 2022 12:32:50 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding CoGroupByKey/GBK as step s12 May 07, 2022 12:32:50 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding CoGroupByKey/ConstructCoGbkResultFn as step s13 May 07, 2022 12:32:50 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding Ungroup and reiterate as step s14 May 07, 2022 12:32:50 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding Collect total bytes as step s15 May 07, 2022 12:32:50 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding Collect end time metrics as step s16 May 07, 2022 12:32:50 PM org.apache.beam.runners.dataflow.DataflowRunner run INFO: Dataflow SDK version: 2.40.0-SNAPSHOT May 07, 2022 12:32:51 PM org.apache.beam.runners.dataflow.DataflowRunner run INFO: To access the Dataflow monitoring console, please navigate to https://console.cloud.google.com/dataflow/jobs/us-central1/2022-05-07_05_32_50-18293218377134002814?project=apache-beam-testing May 07, 2022 12:32:51 PM org.apache.beam.runners.dataflow.DataflowRunner run INFO: Submitted job: 2022-05-07_05_32_50-18293218377134002814 May 07, 2022 12:32:51 PM org.apache.beam.runners.dataflow.DataflowRunner run INFO: To cancel the job using the 'gcloud' tool, run: > gcloud dataflow jobs --project=apache-beam-testing cancel > --region=us-central1 2022-05-07_05_32_50-18293218377134002814 May 07, 2022 12:32:56 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process WARNING: 2022-05-07T12:32:54.377Z: The workflow name is not a valid Cloud Label. Labels applied to Cloud resources (such as GCE Instances) for monitoring will be labeled with this modified job name: load0tests0java170dataflow0v20streaming0cogbk01-jenkins-05-e677. For the best monitoring experience, please name your job with a valid Cloud Label. For details, see: https://cloud.google.com/compute/docs/labeling-resources#restrictions May 07, 2022 12:33:05 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2022-05-07T12:33:03.852Z: Worker configuration: e2-standard-2 in us-central1-b. May 07, 2022 12:33:05 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2022-05-07T12:33:04.691Z: Expanding SplittableParDo operations into optimizable parts. May 07, 2022 12:33:05 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2022-05-07T12:33:04.716Z: Expanding CollectionToSingleton operations into optimizable parts. May 07, 2022 12:33:05 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2022-05-07T12:33:04.786Z: Expanding CoGroupByKey operations into optimizable parts. May 07, 2022 12:33:05 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2022-05-07T12:33:04.935Z: Expanding SplittableProcessKeyed operations into optimizable parts. May 07, 2022 12:33:05 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2022-05-07T12:33:05.134Z: Expanding GroupByKey operations into streaming Read/Write steps May 07, 2022 12:33:07 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2022-05-07T12:33:05.412Z: Lifting ValueCombiningMappingFns into MergeBucketsMappingFns May 07, 2022 12:33:07 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2022-05-07T12:33:05.501Z: Your project already contains 100 Dataflow-created metric descriptors, so new user metrics of the form custom.googleapis.com/* will not be created. However, all user metrics are also available in the metric dataflow.googleapis.com/job/user_counter. If you rely on the custom metrics, you can delete old / unused metric descriptors. See https://developers.google.com/apis-explorer/#p/monitoring/v3/monitoring.projects.metricDescriptors.list and https://developers.google.com/apis-explorer/#p/monitoring/v3/monitoring.projects.metricDescriptors.delete May 07, 2022 12:33:07 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2022-05-07T12:33:05.614Z: Fusing adjacent ParDo, Read, Write, and Flatten operations May 07, 2022 12:33:07 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2022-05-07T12:33:05.650Z: Unzipping flatten CoGroupByKey-Flatten for input CoGroupByKey-MakeUnionTable0-ParMultiDo-ConstructUnionTable-.output May 07, 2022 12:33:07 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2022-05-07T12:33:05.687Z: Fusing unzipped copy of CoGroupByKey/GBK/WriteStream, through flatten CoGroupByKey/Flatten, into producer CoGroupByKey/MakeUnionTable0/ParMultiDo(ConstructUnionTable) May 07, 2022 12:33:07 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2022-05-07T12:33:05.719Z: Fusing consumer CoGroupByKey/GBK/WriteStream into CoGroupByKey/MakeUnionTable1/ParMultiDo(ConstructUnionTable) May 07, 2022 12:33:07 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2022-05-07T12:33:05.753Z: Fusing consumer Read input/ParDo(OutputSingleSource)/ParMultiDo(OutputSingleSource) into Read input/Impulse May 07, 2022 12:33:07 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2022-05-07T12:33:05.786Z: Fusing consumer Read-input-ParDo-UnboundedSourceAsSDFWrapper--ParMultiDo-UnboundedSourceAsSDFWrapper-/PairWithRestriction into Read input/ParDo(OutputSingleSource)/ParMultiDo(OutputSingleSource) May 07, 2022 12:33:07 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2022-05-07T12:33:05.820Z: Fusing consumer Read-input-ParDo-UnboundedSourceAsSDFWrapper--ParMultiDo-UnboundedSourceAsSDFWrapper-/SplitWithSizing into Read-input-ParDo-UnboundedSourceAsSDFWrapper--ParMultiDo-UnboundedSourceAsSDFWrapper-/PairWithRestriction May 07, 2022 12:33:07 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2022-05-07T12:33:05.851Z: Fusing consumer Read input/ParDo(StripIds)/ParMultiDo(StripIds) into Read-input-ParDo-UnboundedSourceAsSDFWrapper--ParMultiDo-UnboundedSourceAsSDFWrapper-/ProcessElementAndRestrictionWithSizing May 07, 2022 12:33:07 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2022-05-07T12:33:05.884Z: Fusing consumer Collect start time metrics (input)/ParMultiDo(TimeMonitor) into Read input/ParDo(StripIds)/ParMultiDo(StripIds) May 07, 2022 12:33:07 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2022-05-07T12:33:05.917Z: Fusing consumer Window.Into()/Window.Assign into Collect start time metrics (input)/ParMultiDo(TimeMonitor) May 07, 2022 12:33:07 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2022-05-07T12:33:05.942Z: Fusing consumer CoGroupByKey/MakeUnionTable0/ParMultiDo(ConstructUnionTable) into Window.Into()/Window.Assign May 07, 2022 12:33:07 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2022-05-07T12:33:05.975Z: Fusing consumer Read co-input/ParDo(OutputSingleSource)/ParMultiDo(OutputSingleSource) into Read co-input/Impulse May 07, 2022 12:33:07 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2022-05-07T12:33:06.008Z: Fusing consumer Read-co-input-ParDo-UnboundedSourceAsSDFWrapper--ParMultiDo-UnboundedSourceAsSDFWrapper-/PairWithRestriction into Read co-input/ParDo(OutputSingleSource)/ParMultiDo(OutputSingleSource) May 07, 2022 12:33:07 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2022-05-07T12:33:06.041Z: Fusing consumer Read-co-input-ParDo-UnboundedSourceAsSDFWrapper--ParMultiDo-UnboundedSourceAsSDFWrapper-/SplitWithSizing into Read-co-input-ParDo-UnboundedSourceAsSDFWrapper--ParMultiDo-UnboundedSourceAsSDFWrapper-/PairWithRestriction May 07, 2022 12:33:07 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2022-05-07T12:33:06.101Z: Fusing consumer Read co-input/ParDo(StripIds)/ParMultiDo(StripIds) into Read-co-input-ParDo-UnboundedSourceAsSDFWrapper--ParMultiDo-UnboundedSourceAsSDFWrapper-/ProcessElementAndRestrictionWithSizing May 07, 2022 12:33:07 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2022-05-07T12:33:06.123Z: Fusing consumer Collect start time metrics (co-input)/ParMultiDo(TimeMonitor) into Read co-input/ParDo(StripIds)/ParMultiDo(StripIds) May 07, 2022 12:33:07 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2022-05-07T12:33:06.166Z: Fusing consumer Window.Into()2/Window.Assign into Collect start time metrics (co-input)/ParMultiDo(TimeMonitor) May 07, 2022 12:33:07 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2022-05-07T12:33:06.199Z: Fusing consumer CoGroupByKey/MakeUnionTable1/ParMultiDo(ConstructUnionTable) into Window.Into()2/Window.Assign May 07, 2022 12:33:07 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2022-05-07T12:33:06.232Z: Fusing consumer CoGroupByKey/GBK/MergeBuckets into CoGroupByKey/GBK/ReadStream May 07, 2022 12:33:07 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2022-05-07T12:33:06.266Z: Fusing consumer CoGroupByKey/ConstructCoGbkResultFn/ParMultiDo(ConstructCoGbkResult) into CoGroupByKey/GBK/MergeBuckets May 07, 2022 12:33:07 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2022-05-07T12:33:06.300Z: Fusing consumer Ungroup and reiterate/ParMultiDo(UngroupAndReiterate) into CoGroupByKey/ConstructCoGbkResultFn/ParMultiDo(ConstructCoGbkResult) May 07, 2022 12:33:07 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2022-05-07T12:33:06.333Z: Fusing consumer Collect total bytes/ParMultiDo(ByteMonitor) into Ungroup and reiterate/ParMultiDo(UngroupAndReiterate) May 07, 2022 12:33:07 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2022-05-07T12:33:06.365Z: Fusing consumer Collect end time metrics/ParMultiDo(TimeMonitor) into Collect total bytes/ParMultiDo(ByteMonitor) May 07, 2022 12:33:07 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2022-05-07T12:33:06.562Z: Running job using Streaming Engine May 07, 2022 12:33:07 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2022-05-07T12:33:06.802Z: Starting 5 ****s in us-central1-b... May 07, 2022 12:33:33 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2022-05-07T12:33:32.472Z: Autoscaling: Raised the number of ****s to 5 so that the pipeline can catch up with its backlog and keep up with its input rate. May 07, 2022 12:34:41 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2022-05-07T12:34:39.607Z: Workers have started successfully. May 07, 2022 1:13:26 PM org.apache.beam.sdk.metrics.MetricsEnvironment getCurrentContainer WARNING: Reporting metrics are not supported in the current execution environment. May 07, 2022 4:01:01 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2022-05-07T16:00:58.218Z: Cancel request is committed for workflow job: 2022-05-07_05_32_50-18293218377134002814. May 07, 2022 4:01:01 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2022-05-07T16:00:59.490Z: Cleaning up. May 07, 2022 4:01:01 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2022-05-07T16:00:59.579Z: Stopping **** pool... May 07, 2022 4:01:01 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2022-05-07T16:00:59.623Z: Stopping **** pool... May 07, 2022 4:01:35 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2022-05-07T16:01:35.077Z: Autoscaling: Reduced the number of ****s to 0 based on low average **** CPU utilization, and the pipeline having sufficiently low backlog and keeping up with input rate. May 07, 2022 4:01:35 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2022-05-07T16:01:35.117Z: Worker pool stopped. May 07, 2022 4:01:45 PM org.apache.beam.runners.dataflow.DataflowPipelineJob logTerminalState INFO: Job 2022-05-07_05_32_50-18293218377134002814 finished with status CANCELLED. Load test results for test (ID): d380543f-25c8-4015-8b36-36a9fa03eb93 and timestamp: 2022-05-07T12:32:45.055000000Z: Metric: Value: dataflow_v2_java17_runtime_sec 12349.746 dataflow_v2_java17_total_bytes_count 3.37999092E10 Exception in thread "main" java.lang.RuntimeException: Invalid job state: CANCELLED. at org.apache.beam.sdk.loadtests.JobFailure.handleFailure(JobFailure.java:51) at org.apache.beam.sdk.loadtests.LoadTest.run(LoadTest.java:139) at org.apache.beam.sdk.loadtests.CoGroupByKeyLoadTest.run(CoGroupByKeyLoadTest.java:62) at org.apache.beam.sdk.loadtests.CoGroupByKeyLoadTest.main(CoGroupByKeyLoadTest.java:157) > Task :sdks:java:testing:load-tests:run FAILED > Task :runners:google-cloud-dataflow-java:cleanUpDockerJavaImages Untagged: us.gcr.io/apache-beam-testing/java-postcommit-it/java:20220507123039 Untagged: us.gcr.io/apache-beam-testing/java-postcommit-it/java@sha256:a201febb4501343aed27f34781031d37aebfe84ff97646ce1d23d15df7f7d15a Tag: [us.gcr.io/apache-beam-testing/java-postcommit-it/java:20220507123039] - referencing digest: [us.gcr.io/apache-beam-testing/java-postcommit-it/java@sha256:a201febb4501343aed27f34781031d37aebfe84ff97646ce1d23d15df7f7d15a] Deleted [[us.gcr.io/apache-beam-testing/java-postcommit-it/java:20220507123039] (referencing [us.gcr.io/apache-beam-testing/java-postcommit-it/java@sha256:a201febb4501343aed27f34781031d37aebfe84ff97646ce1d23d15df7f7d15a])]. Removing untagged image us.gcr.io/apache-beam-testing/java-postcommit-it/java@sha256:a201febb4501343aed27f34781031d37aebfe84ff97646ce1d23d15df7f7d15a Digests: - us.gcr.io/apache-beam-testing/java-postcommit-it/java@sha256:a201febb4501343aed27f34781031d37aebfe84ff97646ce1d23d15df7f7d15a Deleted [us.gcr.io/apache-beam-testing/java-postcommit-it/java@sha256:a201febb4501343aed27f34781031d37aebfe84ff97646ce1d23d15df7f7d15a]. FAILURE: Build failed with an exception. * What went wrong: Execution failed for task ':sdks:java:testing:load-tests:run'. > Process 'command '/usr/lib/jvm/java-8-openjdk-amd64/bin/java'' finished with > non-zero exit value 1 * Try: > Run with --stacktrace option to get the stack trace. > Run with --info or --debug option to get more log output. > Run with --scan to get full insights. * Get more help at https://help.gradle.org Deprecated Gradle features were used in this build, making it incompatible with Gradle 8.0. You can use '--warning-mode all' to show the individual deprecation warnings and determine if they come from your own scripts or plugins. See https://docs.gradle.org/7.4/userguide/command_line_interface.html#sec:command_line_warnings Execution optimizations have been disabled for 1 invalid unit(s) of work during this build to ensure correctness. Please consult deprecation warnings for more details. BUILD FAILED in 3h 31m 26s 110 actionable tasks: 73 executed, 33 from cache, 4 up-to-date Publishing build scan... https://gradle.com/s/icdmjakku6vew Build step 'Invoke Gradle script' changed build result to FAILURE Build step 'Invoke Gradle script' marked build as failure --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
