See
<https://ci-beam.apache.org/job/beam_LoadTests_Java_GBK_Dataflow_V2_Streaming_Java17/138/display/redirect?page=changes>
Changes:
[kevinsijo] Setting up a basic directory
[kevinsijo] Mirroring Python SDK's directory structure
[kerrydc] Adds initial tests
[kevinsijo] 'runners' is the correct directory name
[Pablo Estrada] sketching the core API for JS SDK
[jonathanlui] add .gitignore for node/ts project
[Robert Bradshaw] Worker directory.
[Robert Bradshaw] Fix complile errors with explicit any for callables.
[Robert Bradshaw] Add worker entry point.
[Robert Bradshaw] Add proto generation code.
[Robert Bradshaw] Add generated proto files.
[Robert Bradshaw] Attempts to get ts protos to compile.
[Robert Bradshaw] Exclude ts protos for now.
[Robert Bradshaw] More changes to get ts protos working.
[Robert Bradshaw] Update scripts and config to get protos compiling.
[Robert Bradshaw] Update geenrated files.
[jonathanlui] add build and clean script to compile ts
[Robert Bradshaw] Generate server for loopback worker.
[Robert Bradshaw] Generated grpc servers for loopback.
[Robert Bradshaw] Add typescript formatter.
[Robert Bradshaw] Loopback server (that does nothing).
[Robert Bradshaw] Working server.
[Pablo Estrada] Starting expansion of primitive transforms
[Pablo Estrada] Starting to implement and support standard coders
[Robert Bradshaw] Also generate grpc clients.
[Robert Bradshaw] Basic implementation of worker harness.
[Pablo Estrada] fix the build
[Robert Bradshaw] Add some missing files for worker harness.
[Robert Bradshaw] Refactor operators to use registration.
[jonathanlui] enable ts in mocha
[jonathanlui] update readme
[jonathanlui] --save-dev @types/mocha
[jonathanlui] translate core_test.js to typescript
[Robert Bradshaw] Encapsulate worker service in a class.
[Kenneth Knowles] Port standard_coders_test to typescript (superficially)
[Pablo Estrada] Starting the proto translation of Impulse, ParDo, GBK
[Robert Bradshaw] Add some tests for the worker code.
[Robert Bradshaw] Fixing old lock file error.
[Pablo Estrada] Adding transform names and fixing GBK coder issue
[Robert Bradshaw] npx tsfmt -r src/apache_beam/base.ts
src/apache_beam/transforms/core.ts
[Kenneth Knowles] switch to import style require() statements
[Kenneth Knowles] Add Coder interface using protobufjs classes
[Kenneth Knowles] BytesCoder with some failures
[noreply] Added GeneralObjectCoder and using it as coder for most transforms
(#9)
[Kenneth Knowles] Fix order of arguments to deepEqual
[Kenneth Knowles] Encode expected encoding as binary
[Robert Bradshaw] Refactor API to allow for composites.
[jrmccluskey] Initial setup for automated Java expansion startup
[jrmccluskey] Update exp_service.ts
[Kenneth Knowles] Fix up coder deserialization
[Robert Bradshaw] Simplify GBK coder computation.
[Robert Bradshaw] Remove top-level PValue.
[Pablo Estrada] Make tests green
[Robert Bradshaw] Rename PValueish to PValue.
[jonathanlui] node runner
[jonathanlui] whitespaces
[Robert Bradshaw] Make Runner.run async.
[jonathanlui] bson and fast-deep-equal should not be listed as devdependency
[jrmccluskey] Add basic Dockerfile that starts ExternalWorkerPool
[Robert Bradshaw] Direct runner.
[kevinsijo] Testing expansion service communication
[Robert Bradshaw] Added flatten, assertion checkers.
[Pablo Estrada] progress on basic coders
[Robert Bradshaw] Fixing the build.
[Robert Bradshaw] Cleanup, simplify access.
[Pablo Estrada] Adding limited support for KVCoder and IterableCoder
[Robert Bradshaw] Introduce PipelineContext.
[Robert Bradshaw] Add toProto to all coders.
[Robert Bradshaw] Some work with coders.
[Robert Bradshaw] Remove debug logging.
[Robert Bradshaw] Use coders over data channel.
[Kenneth Knowles] explicitly sequence sub-coder serializations
[Kenneth Knowles] no more need to extend FakeCoder
[Kenneth Knowles] actually advance reader
[Kenneth Knowles] autoformat
[Kenneth Knowles] protobufjs already can write and read signed varints
[Kenneth Knowles] with improved test harness, kv has many more failures
[Kenneth Knowles] read bytescoder from correct position
[Kenneth Knowles] no more fake coders
[Kenneth Knowles] varint examples all work
[Kenneth Knowles] simplify coder value parsing
[Kenneth Knowles] global window coder
[Kenneth Knowles] fix swapEndian32
[Robert Bradshaw] Add P(...) operator.
[kevinsijo] Implementing RowCoder encoding.
[jrmccluskey] remove unused container dir
[kevinsijo] Corrected sorting of encoded positions to reflect an argsort
instead.
[Robert Bradshaw] Populate environments.
[kevinsijo] Implementing RowCoder decoding.
[Kenneth Knowles] preliminary unbounded iterable coder
[Kenneth Knowles] friendlier description of standard coder test case
[Kenneth Knowles] fix test harness; iterable works
[jrmccluskey] first pass at boot.go
[jonathanlui] update package-lock.json
[jonathanlui] make NodeRunner a subclass of Runner
[jonathanlui] add waitUntilFinish interface member
[Pablo Estrada] Adding double coder
[Kenneth Knowles] scaffolding for windowed values
[Pablo Estrada] Adding type information to PColleciton and PTransform
[jonathanlui] fix direct runner
[Pablo Estrada] Adding typing information for DoFns
[Kenneth Knowles] add interval window
[Robert Bradshaw] Export PValue.
[Robert Bradshaw] Add CombineFn interface.
[Robert Bradshaw] Typed flatten.
[jonathanlui] add runAsync method to base.Runner
[Kenneth Knowles] add Long package
[Pablo Estrada] Adding more types. Making PValue typed
[Kenneth Knowles] instant coder draft
[Robert Bradshaw] Return job state from direct runner.
[Kenneth Knowles] type instant = long
[jonathanlui] implement NodeRunner.runPipeline
[Kenneth Knowles] autoformat
[kevinsijo] Completed implementation of basic row coder
[Kenneth Knowles] Fix IntervalWindowCoder, almost
[Kenneth Knowles] fix interval window coder
[Kenneth Knowles] autoformat
[Robert Bradshaw] loopback runner works
[Kenneth Knowles] move core element types into values.ts
[Kenneth Knowles] just build object directly to be cool
[Robert Bradshaw] GBK working on ULR.
[Robert Bradshaw] Async transforms.
[Robert Bradshaw] External transform grpah splicing.
[Kenneth Knowles] progress on windowed value: paneinfo encoding
[Robert Bradshaw] Fix merge.
[Robert Bradshaw] autoformat
[Kenneth Knowles] full windowed value coder
[kerrydc] Updates tests to use correct types, adds generics where needed to
DoFns
[Robert Bradshaw] Add serialization librarires.'
[Robert Bradshaw] Add Split() PTransform, for producing multiple outputs from a
single
[Robert Bradshaw] Schema-encoded external payloads.
[kevinsijo] Adding Schema inference from JSON
[Pablo Estrada] Removing unused directories
[Pablo Estrada] Support for finishBundle and improving typing annotations.
[Pablo Estrada] A base implementation of combiners with GBK/ParDo
[Robert Bradshaw] Fully propagate windowing information in both remote and
direct runner.
[Robert Bradshaw] Make args and kwargs optional for python external transform.
[Robert Bradshaw] Infer schema for external transforms.
[Pablo Estrada] Implementing a custom combine fn as an example. Small fixes
[Robert Bradshaw] Fix missing windowing information in combiners.
[Robert Bradshaw] PostShuffle needn't group by key as that's already done.
[Robert Bradshaw] Guard pre-combine for global window only.
[Robert Bradshaw] WindowInto
[Robert Bradshaw] Fix optional kwargs.
[Robert Bradshaw] A couple of tweaks for js + py
[Robert Bradshaw] Add windowing file.
[Robert Bradshaw] CombineBy transform, stand-alone WordCount.
[Robert Bradshaw] cleanup
[Robert Bradshaw] Actually fix optional external kwargs.
[Robert Bradshaw] Demo2, textio read.
[Robert Bradshaw] Add command lines for starting up the servers.
[Robert Bradshaw] Run prettier on the full codebase.
[Robert Bradshaw] Update deps.
[Pablo Estrada] Adding docstrings for core.ts. Prettier dependency
[Pablo Estrada] Documenting coder interfaces
[Pablo Estrada] Added documentation for a few standard coders
[Robert Bradshaw] Unified grouping and combining.
[Robert Bradshaw] Allow PCollection ids to be lazy.
[Robert Bradshaw] Reorganize module structure.
[Robert Bradshaw] A couple more renames.
[Robert Bradshaw] Simplify.
[Robert Bradshaw] Consolidation.
[Robert Bradshaw] Fix build.
[Robert Bradshaw] Add optional context to ParDo.
[Robert Bradshaw] fixup: iterable coder endian sign issue
[Robert Bradshaw] omit context for map(console.log)
[Robert Bradshaw] Fix ReadFromText coders.
[Robert Bradshaw] Flesh out README with overview and current state.
[noreply] Readme typo
[Robert Bradshaw] Two more TODOs.
[noreply] Add a pointer to the example wordcount to the readme.
[Pablo Estrada] Documenting coders and implementing unknown-length method
[Robert Bradshaw] UIID dependency.
[Robert Bradshaw] Artifact handling.
[Robert Bradshaw] Properly wait on data channel for bundle completion.
[Robert Bradshaw] Automatic java expansion service startup.
[Robert Bradshaw] Process promises.
[Robert Bradshaw] Implement side inputs.
[Robert Bradshaw] Cleanup.
[Robert Bradshaw] Put complex constext stuff in its own file.
[Robert Bradshaw] Rename BoundedWindow to just Window.
[Robert Bradshaw] Alternative splitter class.
[Pablo Estrada] Documenting internal functions
[Robert Bradshaw] Take a pass clarifying the TODOs.
[Robert Bradshaw] Sql transform wrapper.
[Robert Bradshaw] Incorporate some feedback into the TODOs.
[Robert Bradshaw] More TODOs.
[Robert Bradshaw] Remove app placeholder.
[Robert Bradshaw] Apache license headers.
[Robert Bradshaw] More TODOs
[jankuehle] Suggestions for TypeScript todos
[dannymccormick] Add actions for typescript sdk
[dannymccormick] Fix test command
[noreply] Add missing version
[dannymccormick] Fix codecovTest command
[noreply] Only do prettier check on linux
[noreply] Only get codecov on linux
[Robert Bradshaw] Resolve some comments.
[Robert Bradshaw] Fix compile errors.
[Robert Bradshaw] Prettier.
[Robert Bradshaw] Re-order expandInternal arguments pending unification.
[Robert Bradshaw] More consistent and stricter PTransform naming.
[Robert Bradshaw] Notes on explicit, if less idiomatic, use of classes.
[Robert Bradshaw] Let DoFn be an interface rather than a class.
[Robert Bradshaw] Provide DoFn context to start and finish bundle.
[Robert Bradshaw] Optional promise code simplification.
[Robert Bradshaw] Cleanup todos.
[Robert Bradshaw] Avoid any type where not needed.
[Robert Bradshaw] Apache RAT excludes for typescript.
[Robert Bradshaw] Remove empty READMEs.
[Robert Bradshaw] Add licences statement to readme files.
[Robert Bradshaw] More RAT fixes.
[Robert Bradshaw] Another unsupported coder.
[Robert Bradshaw] Remove debugging code.
[noreply] Fix automatic naming with code coverage.
[Robert Bradshaw] Coders cleanup.
[Robert Bradshaw] Add tests for RowCoder.
[Robert Bradshaw] Normalize capitalization, comments.
[Robert Bradshaw] Install typescript closure packages.
[Robert Bradshaw] npm audit fix
[Robert Bradshaw] Move more imports out of base.
[Robert Bradshaw] Changes needed to compile with ts closure plugin.
[Robert Bradshaw] Use ttsc and ts-closure-transform plugin.
[Robert Bradshaw] Serialization registration to actually get serialization
working.
[Robert Bradshaw] Container images working on local runner.
[Robert Bradshaw] Add a portable job server that proxies the Dataflow backend.
(#17189)
[Robert Bradshaw] Improvements to dataflow job service for non-Python jobs.
[Robert Bradshaw] Get dataflow working.
[Robert Bradshaw] User friendly pipeline options.
[Robert Bradshaw] Less classes, more functions.
[Robert Bradshaw] Add new nullable standard coder.
[Robert Bradshaw] Make Apache Rat happy.
[Robert Bradshaw] Disable broken codecov.
[Robert Bradshaw] Remove last uses of base.ts.
[Robert Bradshaw] Remove unneedd file.
[Robert Bradshaw] Remove more uneeded/unused files.
[Robert Bradshaw] Cleanup tests.
[Robert Bradshaw] Minor cleanups to coder tests.
[noreply] Quote pip install package name
[noreply] [BEAM-14374] Fix module import error in FullyQualifiedNamedTransform
[Robert Bradshaw] Addressing issues from the review.
[noreply] Apply suggestions from code review.
[Robert Bradshaw] Post-merge fixes.
[dannymccormick] Delete tags.go
[Robert Bradshaw] Update tests to use our actual serialization libraries.
[Robert Bradshaw] Another pass at TODOs, removing finished items.
[Heejong Lee] [BEAM-14146] Python Streaming job failing to drain with
BigQueryIO write
[Kenneth Knowles] Add parameter for service account impersonation in GCP
credentials
[Heejong Lee] add test
[noreply] Merge pull request #17490 from [BEAM-14370] [Website] Add new page
about
[noreply] [BEAM-14332] Refactored cluster management for Flink on Dataproc
[noreply] [BEAM-13988] Update mtime to use time.UnixMilli() calls (#17578)
[noreply] Fixing patching error on missing dependencies (#17564)
[noreply] Merge pull request #17517 from [BEAM-14383] Improve "FailedRows"
errors
[Heejong Lee] add test without mock
------------------------------------------
[...truncated 66.22 KB...]
INFO: 2022-05-07T13:48:56.735Z: Autoscaling: Raised the number of ****s to 5 so
that the pipeline can catch up with its backlog and keep up with its input rate.
May 07, 2022 1:50:30 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
SEVERE: 2022-05-07T13:50:27.360Z: generic::aborted: SDK harness sdk-0-0
disconnected.
generic::aborted: SDK harness sdk-0-0 disconnected.
passed through:
==>
dist_proc/dax/workflow/****/fnapi_service_impl.cc:922
May 07, 2022 1:51:21 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
SEVERE: 2022-05-07T13:51:20.616Z: generic::aborted: SDK harness sdk-0-0
disconnected.
generic::aborted: SDK harness sdk-0-0 disconnected.
passed through:
==>
dist_proc/dax/workflow/****/fnapi_service_impl.cc:922
May 07, 2022 1:52:22 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
SEVERE: 2022-05-07T13:52:20.145Z: generic::aborted: SDK harness sdk-0-0
disconnected.
passed through:
==>
dist_proc/dax/workflow/****/monitoring_info_view_creator.cc:68
generic::aborted: SDK harness sdk-0-0 disconnected.
generic::aborted: SDK harness sdk-0-0 disconnected.
passed through:
==>
dist_proc/dax/workflow/****/fnapi_service_impl.cc:922
May 07, 2022 1:53:49 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
SEVERE: 2022-05-07T13:53:46.998Z: generic::aborted: SDK harness sdk-0-0
disconnected.
passed through:
==>
dist_proc/dax/workflow/****/fnapi_service_impl.cc:922
generic::aborted: SDK harness sdk-0-0 disconnected.
May 07, 2022 1:55:50 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
SEVERE: 2022-05-07T13:55:48.017Z: generic::cancelled: Data channel closed,
unable to receive additional data from SDK sdk-0-0
passed through:
==>
dist_proc/dax/workflow/****/fnapi_service_impl.cc:922
generic::cancelled: Data channel closed, unable to receive additional data from
SDK sdk-0-0
May 07, 2022 1:57:59 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
SEVERE: 2022-05-07T13:57:56.951Z: generic::cancelled: Data channel closed,
unable to receive additional data from SDK sdk-0-0
passed through:
==>
dist_proc/dax/workflow/****/monitoring_info_view_creator.cc:68
generic::cancelled: Data channel closed, unable to receive additional data from
SDK sdk-0-0
passed through:
==>
dist_proc/dax/workflow/****/fnapi_service_impl.cc:922
generic::cancelled: Data channel closed, unable to receive additional data from
SDK sdk-0-0
May 07, 2022 1:58:02 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-05-07T13:58:00.976Z: Autoscaling: Raised the number of ****s to 5 so
that the pipeline can catch up with its backlog and keep up with its input rate.
May 07, 2022 2:00:05 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-05-07T14:00:03.611Z: Autoscaling: Raised the number of ****s to 5 so
that the pipeline can catch up with its backlog and keep up with its input rate.
May 07, 2022 2:01:03 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-05-07T14:01:02.072Z: Autoscaling: Raised the number of ****s to 5 so
that the pipeline can catch up with its backlog and keep up with its input rate.
May 07, 2022 2:01:20 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
SEVERE: 2022-05-07T14:01:18.665Z: generic::aborted: SDK harness sdk-0-0
disconnected.
passed through:
==>
dist_proc/dax/workflow/****/fnapi_service_impl.cc:922
generic::aborted: SDK harness sdk-0-0 disconnected.
May 07, 2022 2:02:42 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
SEVERE: 2022-05-07T14:02:42.508Z: generic::cancelled: Data channel closed,
unable to receive additional data from SDK sdk-0-0
generic::cancelled: Data channel closed, unable to receive additional data from
SDK sdk-0-0
passed through:
==>
dist_proc/dax/workflow/****/fnapi_service_impl.cc:922
May 07, 2022 2:03:16 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
SEVERE: 2022-05-07T14:03:14.828Z: generic::aborted: SDK harness sdk-0-0
disconnected.
passed through:
==>
dist_proc/dax/workflow/****/fnapi_service_impl.cc:922
generic::aborted: SDK harness sdk-0-0 disconnected.
May 07, 2022 2:07:06 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-05-07T14:07:05.720Z: Autoscaling: Raised the number of ****s to 5 so
that the pipeline can catch up with its backlog and keep up with its input rate.
May 07, 2022 2:07:38 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
SEVERE: 2022-05-07T14:07:37.511Z: generic::aborted: SDK harness sdk-0-0
disconnected.
generic::aborted: SDK harness sdk-0-0 disconnected.
passed through:
==>
dist_proc/dax/workflow/****/fnapi_service_impl.cc:922
May 07, 2022 2:10:09 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-05-07T14:10:08.780Z: Autoscaling: Raised the number of ****s to 5 so
that the pipeline can catch up with its backlog and keep up with its input rate.
May 07, 2022 2:12:09 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-05-07T14:12:07.676Z: Autoscaling: Raised the number of ****s to 5 so
that the pipeline can catch up with its backlog and keep up with its input rate.
May 07, 2022 2:13:21 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-05-07T14:13:20.386Z: Autoscaling: Raised the number of ****s to 5 so
that the pipeline can catch up with its backlog and keep up with its input rate.
May 07, 2022 2:14:59 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
SEVERE: 2022-05-07T14:14:58.243Z: generic::aborted: SDK harness sdk-0-0
disconnected.
passed through:
==>
dist_proc/dax/workflow/****/fnapi_service_impl.cc:922
generic::aborted: SDK harness sdk-0-0 disconnected.
May 07, 2022 2:18:14 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-05-07T14:18:14.036Z: Autoscaling: Raised the number of ****s to 5 so
that the pipeline can catch up with its backlog and keep up with its input rate.
May 07, 2022 2:19:14 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-05-07T14:19:12.527Z: Autoscaling: Raised the number of ****s to 5 so
that the pipeline can catch up with its backlog and keep up with its input rate.
May 07, 2022 2:20:49 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
SEVERE: 2022-05-07T14:20:47.234Z: generic::aborted: SDK harness sdk-0-0
disconnected.
generic::aborted: SDK harness sdk-0-0 disconnected.
passed through:
==>
dist_proc/dax/workflow/****/fnapi_service_impl.cc:922
May 07, 2022 2:23:18 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-05-07T14:23:15.996Z: Autoscaling: Raised the number of ****s to 5 so
that the pipeline can catch up with its backlog and keep up with its input rate.
May 07, 2022 2:26:29 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-05-07T14:26:28.913Z: Autoscaling: Raised the number of ****s to 5 so
that the pipeline can catch up with its backlog and keep up with its input rate.
May 07, 2022 2:27:19 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-05-07T14:27:17.827Z: Autoscaling: Raised the number of ****s to 5 so
that the pipeline can catch up with its backlog and keep up with its input rate.
May 07, 2022 2:28:21 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-05-07T14:28:20.299Z: Autoscaling: Raised the number of ****s to 5 so
that the pipeline can catch up with its backlog and keep up with its input rate.
May 07, 2022 2:35:23 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-05-07T14:35:23.812Z: Autoscaling: Raised the number of ****s to 5 so
that the pipeline can catch up with its backlog and keep up with its input rate.
May 07, 2022 2:39:23 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-05-07T14:39:23.215Z: Autoscaling: Raised the number of ****s to 5 so
that the pipeline can catch up with its backlog and keep up with its input rate.
May 07, 2022 2:44:27 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-05-07T14:44:26.056Z: Autoscaling: Raised the number of ****s to 5 so
that the pipeline can catch up with its backlog and keep up with its input rate.
May 07, 2022 2:50:30 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-05-07T14:50:29.851Z: Autoscaling: Raised the number of ****s to 5 so
that the pipeline can catch up with its backlog and keep up with its input rate.
May 07, 2022 2:53:27 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-05-07T14:53:27.408Z: Autoscaling: Raised the number of ****s to 5 so
that the pipeline can catch up with its backlog and keep up with its input rate.
May 07, 2022 2:54:30 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-05-07T14:54:29.895Z: Autoscaling: Raised the number of ****s to 5 so
that the pipeline can catch up with its backlog and keep up with its input rate.
May 07, 2022 3:03:35 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-05-07T15:03:34.024Z: Autoscaling: Raised the number of ****s to 5 so
that the pipeline can catch up with its backlog and keep up with its input rate.
May 07, 2022 3:12:35 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-05-07T15:12:33.398Z: Autoscaling: Raised the number of ****s to 5 so
that the pipeline can catch up with its backlog and keep up with its input rate.
May 07, 2022 3:18:38 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-05-07T15:18:36.683Z: Autoscaling: Raised the number of ****s to 5 so
that the pipeline can catch up with its backlog and keep up with its input rate.
May 07, 2022 3:21:40 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-05-07T15:21:39.486Z: Autoscaling: Raised the number of ****s to 5 so
that the pipeline can catch up with its backlog and keep up with its input rate.
May 07, 2022 3:28:39 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-05-07T15:28:38.286Z: Autoscaling: Raised the number of ****s to 5 so
that the pipeline can catch up with its backlog and keep up with its input rate.
May 07, 2022 3:33:41 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-05-07T15:33:41.352Z: Autoscaling: Raised the number of ****s to 5 so
that the pipeline can catch up with its backlog and keep up with its input rate.
May 07, 2022 3:37:46 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-05-07T15:37:44.096Z: Autoscaling: Raised the number of ****s to 5 so
that the pipeline can catch up with its backlog and keep up with its input rate.
May 07, 2022 3:38:47 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-05-07T15:38:46.425Z: Autoscaling: Raised the number of ****s to 5 so
that the pipeline can catch up with its backlog and keep up with its input rate.
May 07, 2022 3:41:47 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-05-07T15:41:45.601Z: Autoscaling: Raised the number of ****s to 5 so
that the pipeline can catch up with its backlog and keep up with its input rate.
May 07, 2022 3:45:49 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-05-07T15:45:48.539Z: Autoscaling: Raised the number of ****s to 5 so
that the pipeline can catch up with its backlog and keep up with its input rate.
May 07, 2022 3:46:50 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-05-07T15:46:50.261Z: Autoscaling: Raised the number of ****s to 5 so
that the pipeline can catch up with its backlog and keep up with its input rate.
May 07, 2022 3:47:53 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-05-07T15:47:52.808Z: Autoscaling: Raised the number of ****s to 5 so
that the pipeline can catch up with its backlog and keep up with its input rate.
May 07, 2022 4:01:02 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-05-07T16:01:02.436Z: Cancel request is committed for workflow job:
2022-05-07_05_59_43-13010494145204206641.
May 07, 2022 4:01:02 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-05-07T16:01:02.519Z: Cleaning up.
May 07, 2022 4:01:02 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-05-07T16:01:02.592Z: Stopping **** pool...
May 07, 2022 4:01:02 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-05-07T16:01:02.649Z: Stopping **** pool...
May 07, 2022 4:03:30 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-05-07T16:03:28.863Z: Autoscaling: Reduced the number of ****s to 0
based on low average **** CPU utilization, and the pipeline having sufficiently
low backlog and keeping up with input rate.
May 07, 2022 4:03:30 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-05-07T16:03:28.904Z: Worker pool stopped.
May 07, 2022 4:03:36 PM org.apache.beam.runners.dataflow.DataflowPipelineJob
logTerminalState
INFO: Job 2022-05-07_05_59_43-13010494145204206641 finished with status
CANCELLED.
Load test results for test (ID): 5fb0eb4b-4be2-49aa-99fb-34f223327da5 and
timestamp: 2022-05-07T12:59:37.563000000Z:
Metric: Value:
dataflow_v2_java17_runtime_sec 10596.19
dataflow_v2_java17_total_bytes_count 4.94593268E9
Exception in thread "main" java.lang.RuntimeException: Invalid job state:
CANCELLED.
at
org.apache.beam.sdk.loadtests.JobFailure.handleFailure(JobFailure.java:51)
at org.apache.beam.sdk.loadtests.LoadTest.run(LoadTest.java:139)
at
org.apache.beam.sdk.loadtests.GroupByKeyLoadTest.run(GroupByKeyLoadTest.java:57)
at
org.apache.beam.sdk.loadtests.GroupByKeyLoadTest.main(GroupByKeyLoadTest.java:131)
> Task :sdks:java:testing:load-tests:run FAILED
> Task :runners:google-cloud-dataflow-java:cleanUpDockerJavaImages
Untagged: us.gcr.io/apache-beam-testing/java-postcommit-it/java:20220507125738
Untagged:
us.gcr.io/apache-beam-testing/java-postcommit-it/java@sha256:fdbf2a8918c858fb9a8bd6a2f42ead14689184860aac62bd8d02049b0ce9d6ec
Deleted: sha256:54efb149cf1199557a9f16658936089a7d5d6b367218830fa0ec26063ddcb793
Deleted: sha256:57b2bba1c8b170ba391a4d116bb5a65555c9bd05a00a55e4f2a275b34b19409a
Deleted: sha256:90d2a12beb84418c57f66a8061a0656b6bf21749e02acf6f39d9e195ffad74e2
Deleted: sha256:479716f3fe7a43e3faa8d743ab0d520872f53409da5374051491578966e10f75
Deleted: sha256:491af3b7a6e485d246d64fbaa7d62f7e488f5a1af12fe17524f7bb4778cd44c3
Deleted: sha256:b11ad2087bcf2915730b6546aba2e2804cca9a05bbef918e26e46043b07e63bb
Deleted: sha256:d33f4b1834fa880a843c9b96f376db1d896f78cfb4df321723ceb8bbaa5af373
Deleted: sha256:3a3ffb80f007c05f2b4ede1549ac11608bb2f51bd3fad5e40bd2b5692a490ca9
Deleted: sha256:14564eb7756cf5f4ec195cc0ee6d6da859477ed6877b1490c004cf68567469d6
Deleted: sha256:da9efb87033da77edc0dcf389d515a5d0e9845f9cfd6cdc51a7867f13b10a2e0
Deleted: sha256:49cceb9eb6861a3d51655e556c47b9cfcd4fc0bda0422911e194fc9a938060d2
Deleted: sha256:e9cd00b526c44dfe00c2b4f7dcd72e4b2504e1f7e4b24d1bc0450fb99761e37e
Deleted: sha256:e3c5733544de158013894ac83c5a1bd9e44c1465dc0b91037fde06a82e007ec7
Deleted: sha256:241222999c1f0ed06a49bb9fcbb2692b6d3333c982611e56bd2cdbe72f07f082
Deleted: sha256:93bc297173f992bf4e20ff3a721f2af55b3c7a69096ff04c4dbfc334798173bd
Deleted: sha256:fda786145476ed0c1a7aeacce9be0bb46219dbd144302e7ed62d53e28b784ae1
Deleted: sha256:7713b3426c7005fe98afb3c5f6b253e64e507c5dbd9db18cb0fee902b5bf1f5e
Deleted: sha256:066fa07c43ba0c41551830059a725ffa12cb587bdfb0b5908004ef4715a816dc
Deleted: sha256:364653f8d24d71196ef426b9e936a5eb8a8cea2b241516e8c466c26c67aae993
Deleted: sha256:039265de93a7b7e4e43b4e421874d92ab4f3bba7b44ba8da1c4651e52025047d
Deleted: sha256:1ceca7bde0cdc1148cccd0ddcc18806a679050bf456d070d410252cfc61bd039
Deleted: sha256:59d2a05520b0e73a742276dd336a85dc3a8bac29f083f06298bc929105650ff3
Deleted: sha256:b242e3e9e8caf93f9b5b58558e6a684170e8e795eb63d2a1bf4b0f5e8a4f1c31
Deleted: sha256:c14758172817aa36592b0f71c28e5648ca7b56421220d294aaee1d32ddf82b95
Deleted: sha256:0337c30c88ec25b6c1a8076467a56604e546fa8e9c549eb73b720f0f99778f1f
Deleted: sha256:50db629c4673bd593c252c9372a3d4f2d0f0f9dc3c53a06440a3d3d4a4e38242
Deleted: sha256:dc590e3920cf09814281554e9769de1d53ae2e7715894f00c7c91cfacae88d7a
Deleted: sha256:73f99e58421787e23999f7edc739e42315ada16268cedcfe848cd7dfb1ed4ee0
Deleted: sha256:273e674e45a99b342e716c9cfe0b1c1c9c6a3e351462c003538cb4594145da92
Deleted: sha256:90abfaa0a76d5875b2b0e5e90d920929f2789fc5fa87e53b9aabe155014189b8
Deleted: sha256:27f30890aef2e56e8463dc0a8dfec90757a8474a16dacb03a2575047d90a0a69
Deleted: sha256:0b688e0ed7fe345ce0c55e9ff9fa312a0a6eddc7ddf886909d9b1892112bd292
Deleted: sha256:799581473abf5747e2d6fd1364f520402b7a74376ec3d85cd4ac19339dd30169
Deleted: sha256:5ecf532ec27206213060b630c3f69b23167d83d49872e9359ce700a39fe2f9ba
Tag: [us.gcr.io/apache-beam-testing/java-postcommit-it/java:20220507125738]
- referencing digest:
[us.gcr.io/apache-beam-testing/java-postcommit-it/java@sha256:fdbf2a8918c858fb9a8bd6a2f42ead14689184860aac62bd8d02049b0ce9d6ec]
Deleted [[us.gcr.io/apache-beam-testing/java-postcommit-it/java:20220507125738]
(referencing
[us.gcr.io/apache-beam-testing/java-postcommit-it/java@sha256:fdbf2a8918c858fb9a8bd6a2f42ead14689184860aac62bd8d02049b0ce9d6ec])].
Removing untagged image
us.gcr.io/apache-beam-testing/java-postcommit-it/java@sha256:fdbf2a8918c858fb9a8bd6a2f42ead14689184860aac62bd8d02049b0ce9d6ec
Digests:
-
us.gcr.io/apache-beam-testing/java-postcommit-it/java@sha256:fdbf2a8918c858fb9a8bd6a2f42ead14689184860aac62bd8d02049b0ce9d6ec
Deleted
[us.gcr.io/apache-beam-testing/java-postcommit-it/java@sha256:fdbf2a8918c858fb9a8bd6a2f42ead14689184860aac62bd8d02049b0ce9d6ec].
Removing untagged image
us.gcr.io/apache-beam-testing/java-postcommit-it/java@sha256:96ac0d2c40c3fc3c43c88aece057673e3963bca27725f88e41f47db7d5faae3e
Digests:
-
us.gcr.io/apache-beam-testing/java-postcommit-it/java@sha256:96ac0d2c40c3fc3c43c88aece057673e3963bca27725f88e41f47db7d5faae3e
Deleted
[us.gcr.io/apache-beam-testing/java-postcommit-it/java@sha256:96ac0d2c40c3fc3c43c88aece057673e3963bca27725f88e41f47db7d5faae3e].
FAILURE: Build failed with an exception.
* What went wrong:
Execution failed for task ':sdks:java:testing:load-tests:run'.
> Process 'command '/usr/lib/jvm/java-8-openjdk-amd64/bin/java'' finished with
> non-zero exit value 1
* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Run with --scan to get full insights.
* Get more help at https://help.gradle.org
Deprecated Gradle features were used in this build, making it incompatible with
Gradle 8.0.
You can use '--warning-mode all' to show the individual deprecation warnings
and determine if they come from your own scripts or plugins.
See
https://docs.gradle.org/7.4/userguide/command_line_interface.html#sec:command_line_warnings
Execution optimizations have been disabled for 1 invalid unit(s) of work during
this build to ensure correctness.
Please consult deprecation warnings for more details.
BUILD FAILED in 3h 6m 22s
110 actionable tasks: 73 executed, 33 from cache, 4 up-to-date
Publishing build scan...
https://gradle.com/s/4tdlaj7ikpeze
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
