See <https://ci-beam.apache.org/job/beam_PostCommit_XVR_JavaUsingPython_Dataflow/395/display/redirect?page=changes>
Changes: [kevinsijo] Setting up a basic directory [kevinsijo] Mirroring Python SDK's directory structure [kerrydc] Adds initial tests [kevinsijo] 'runners' is the correct directory name [Pablo Estrada] sketching the core API for JS SDK [jonathanlui] add .gitignore for node/ts project [Robert Bradshaw] Worker directory. [Robert Bradshaw] Fix complile errors with explicit any for callables. [Robert Bradshaw] Add worker entry point. [Robert Bradshaw] Add proto generation code. [Robert Bradshaw] Add generated proto files. [Robert Bradshaw] Attempts to get ts protos to compile. [Robert Bradshaw] Exclude ts protos for now. [Robert Bradshaw] More changes to get ts protos working. [Robert Bradshaw] Update scripts and config to get protos compiling. [Robert Bradshaw] Update geenrated files. [jonathanlui] add build and clean script to compile ts [Robert Bradshaw] Generate server for loopback worker. [Robert Bradshaw] Generated grpc servers for loopback. [Robert Bradshaw] Add typescript formatter. [Robert Bradshaw] Loopback server (that does nothing). [Robert Bradshaw] Working server. [Pablo Estrada] Starting expansion of primitive transforms [Pablo Estrada] Starting to implement and support standard coders [Robert Bradshaw] Also generate grpc clients. [Robert Bradshaw] Basic implementation of worker harness. [Pablo Estrada] fix the build [Robert Bradshaw] Add some missing files for worker harness. [Robert Bradshaw] Refactor operators to use registration. [jonathanlui] enable ts in mocha [jonathanlui] update readme [jonathanlui] --save-dev @types/mocha [jonathanlui] translate core_test.js to typescript [Robert Bradshaw] Encapsulate worker service in a class. [Kenneth Knowles] Port standard_coders_test to typescript (superficially) [Pablo Estrada] Starting the proto translation of Impulse, ParDo, GBK [Robert Bradshaw] Add some tests for the worker code. [Robert Bradshaw] Fixing old lock file error. [Pablo Estrada] Adding transform names and fixing GBK coder issue [Robert Bradshaw] npx tsfmt -r src/apache_beam/base.ts src/apache_beam/transforms/core.ts [Kenneth Knowles] switch to import style require() statements [Kenneth Knowles] Add Coder interface using protobufjs classes [Kenneth Knowles] BytesCoder with some failures [noreply] Added GeneralObjectCoder and using it as coder for most transforms (#9) [Kenneth Knowles] Fix order of arguments to deepEqual [Kenneth Knowles] Encode expected encoding as binary [Robert Bradshaw] Refactor API to allow for composites. [jrmccluskey] Initial setup for automated Java expansion startup [jrmccluskey] Update exp_service.ts [Kenneth Knowles] Fix up coder deserialization [Robert Bradshaw] Simplify GBK coder computation. [Robert Bradshaw] Remove top-level PValue. [Pablo Estrada] Make tests green [Robert Bradshaw] Rename PValueish to PValue. [jonathanlui] node runner [jonathanlui] whitespaces [Robert Bradshaw] Make Runner.run async. [jonathanlui] bson and fast-deep-equal should not be listed as devdependency [jrmccluskey] Add basic Dockerfile that starts ExternalWorkerPool [Robert Bradshaw] Direct runner. [kevinsijo] Testing expansion service communication [Robert Bradshaw] Added flatten, assertion checkers. [Pablo Estrada] progress on basic coders [Robert Bradshaw] Fixing the build. [Robert Bradshaw] Cleanup, simplify access. [Pablo Estrada] Adding limited support for KVCoder and IterableCoder [Robert Bradshaw] Introduce PipelineContext. [Robert Bradshaw] Add toProto to all coders. [Robert Bradshaw] Some work with coders. [Robert Bradshaw] Remove debug logging. [Robert Bradshaw] Use coders over data channel. [Kenneth Knowles] explicitly sequence sub-coder serializations [Kenneth Knowles] no more need to extend FakeCoder [Kenneth Knowles] actually advance reader [Kenneth Knowles] autoformat [Kenneth Knowles] protobufjs already can write and read signed varints [Kenneth Knowles] with improved test harness, kv has many more failures [Kenneth Knowles] read bytescoder from correct position [Kenneth Knowles] no more fake coders [Kenneth Knowles] varint examples all work [Kenneth Knowles] simplify coder value parsing [Kenneth Knowles] global window coder [Kenneth Knowles] fix swapEndian32 [Robert Bradshaw] Add P(...) operator. [kevinsijo] Implementing RowCoder encoding. [jrmccluskey] remove unused container dir [kevinsijo] Corrected sorting of encoded positions to reflect an argsort instead. [Robert Bradshaw] Populate environments. [kevinsijo] Implementing RowCoder decoding. [Kenneth Knowles] preliminary unbounded iterable coder [Kenneth Knowles] friendlier description of standard coder test case [Kenneth Knowles] fix test harness; iterable works [jrmccluskey] first pass at boot.go [jonathanlui] update package-lock.json [jonathanlui] make NodeRunner a subclass of Runner [jonathanlui] add waitUntilFinish interface member [Pablo Estrada] Adding double coder [Kenneth Knowles] scaffolding for windowed values [Pablo Estrada] Adding type information to PColleciton and PTransform [jonathanlui] fix direct runner [Pablo Estrada] Adding typing information for DoFns [Kenneth Knowles] add interval window [Robert Bradshaw] Export PValue. [Robert Bradshaw] Add CombineFn interface. [Robert Bradshaw] Typed flatten. [jonathanlui] add runAsync method to base.Runner [Kenneth Knowles] add Long package [Pablo Estrada] Adding more types. Making PValue typed [Kenneth Knowles] instant coder draft [Robert Bradshaw] Return job state from direct runner. [Kenneth Knowles] type instant = long [jonathanlui] implement NodeRunner.runPipeline [Kenneth Knowles] autoformat [kevinsijo] Completed implementation of basic row coder [Kenneth Knowles] Fix IntervalWindowCoder, almost [Kenneth Knowles] fix interval window coder [Kenneth Knowles] autoformat [Robert Bradshaw] loopback runner works [Kenneth Knowles] move core element types into values.ts [Kenneth Knowles] just build object directly to be cool [Robert Bradshaw] GBK working on ULR. [Robert Bradshaw] Async transforms. [Robert Bradshaw] External transform grpah splicing. [Kenneth Knowles] progress on windowed value: paneinfo encoding [Robert Bradshaw] Fix merge. [Robert Bradshaw] autoformat [Kenneth Knowles] full windowed value coder [kerrydc] Updates tests to use correct types, adds generics where needed to DoFns [Robert Bradshaw] Add serialization librarires.' [Robert Bradshaw] Add Split() PTransform, for producing multiple outputs from a single [Robert Bradshaw] Schema-encoded external payloads. [kevinsijo] Adding Schema inference from JSON [Pablo Estrada] Removing unused directories [Pablo Estrada] Support for finishBundle and improving typing annotations. [Pablo Estrada] A base implementation of combiners with GBK/ParDo [Robert Bradshaw] Fully propagate windowing information in both remote and direct runner. [Robert Bradshaw] Make args and kwargs optional for python external transform. [Robert Bradshaw] Infer schema for external transforms. [Pablo Estrada] Implementing a custom combine fn as an example. Small fixes [Robert Bradshaw] Fix missing windowing information in combiners. [Robert Bradshaw] PostShuffle needn't group by key as that's already done. [Robert Bradshaw] Guard pre-combine for global window only. [Robert Bradshaw] WindowInto [Robert Bradshaw] Fix optional kwargs. [Robert Bradshaw] A couple of tweaks for js + py [Robert Bradshaw] Add windowing file. [Robert Bradshaw] CombineBy transform, stand-alone WordCount. [Robert Bradshaw] cleanup [Robert Bradshaw] Actually fix optional external kwargs. [Robert Bradshaw] Demo2, textio read. [Robert Bradshaw] Add command lines for starting up the servers. [Robert Bradshaw] Run prettier on the full codebase. [Robert Bradshaw] Update deps. [Pablo Estrada] Adding docstrings for core.ts. Prettier dependency [Pablo Estrada] Documenting coder interfaces [Pablo Estrada] Added documentation for a few standard coders [Robert Bradshaw] Unified grouping and combining. [Robert Bradshaw] Allow PCollection ids to be lazy. [Robert Bradshaw] Reorganize module structure. [Robert Bradshaw] A couple more renames. [Robert Bradshaw] Simplify. [Robert Bradshaw] Consolidation. [Robert Bradshaw] Fix build. [Robert Bradshaw] Add optional context to ParDo. [Robert Bradshaw] fixup: iterable coder endian sign issue [Robert Bradshaw] omit context for map(console.log) [Robert Bradshaw] Fix ReadFromText coders. [Robert Bradshaw] Flesh out README with overview and current state. [noreply] Readme typo [Robert Bradshaw] Two more TODOs. [noreply] Add a pointer to the example wordcount to the readme. [Pablo Estrada] Documenting coders and implementing unknown-length method [Robert Bradshaw] UIID dependency. [Robert Bradshaw] Artifact handling. [Robert Bradshaw] Properly wait on data channel for bundle completion. [Robert Bradshaw] Automatic java expansion service startup. [Robert Bradshaw] Process promises. [Robert Bradshaw] Implement side inputs. [Robert Bradshaw] Cleanup. [Robert Bradshaw] Put complex constext stuff in its own file. [Robert Bradshaw] Rename BoundedWindow to just Window. [Robert Bradshaw] Alternative splitter class. [Pablo Estrada] Documenting internal functions [Robert Bradshaw] Take a pass clarifying the TODOs. [Robert Bradshaw] Sql transform wrapper. [Robert Bradshaw] Incorporate some feedback into the TODOs. [Robert Bradshaw] More TODOs. [Robert Bradshaw] Remove app placeholder. [Robert Bradshaw] Apache license headers. [Robert Bradshaw] More TODOs [jankuehle] Suggestions for TypeScript todos [dannymccormick] Add actions for typescript sdk [dannymccormick] Fix test command [noreply] Add missing version [dannymccormick] Fix codecovTest command [noreply] Only do prettier check on linux [noreply] Only get codecov on linux [Robert Bradshaw] Resolve some comments. [Robert Bradshaw] Fix compile errors. [Robert Bradshaw] Prettier. [Robert Bradshaw] Re-order expandInternal arguments pending unification. [Robert Bradshaw] More consistent and stricter PTransform naming. [Robert Bradshaw] Notes on explicit, if less idiomatic, use of classes. [Robert Bradshaw] Let DoFn be an interface rather than a class. [Robert Bradshaw] Provide DoFn context to start and finish bundle. [Robert Bradshaw] Optional promise code simplification. [Robert Bradshaw] Cleanup todos. [Robert Bradshaw] Avoid any type where not needed. [Robert Bradshaw] Apache RAT excludes for typescript. [Robert Bradshaw] Remove empty READMEs. [Robert Bradshaw] Add licences statement to readme files. [Robert Bradshaw] More RAT fixes. [Robert Bradshaw] Another unsupported coder. [Robert Bradshaw] Remove debugging code. [noreply] Fix automatic naming with code coverage. [Robert Bradshaw] Coders cleanup. [Robert Bradshaw] Add tests for RowCoder. [Robert Bradshaw] Normalize capitalization, comments. [Robert Bradshaw] Install typescript closure packages. [Robert Bradshaw] npm audit fix [Robert Bradshaw] Move more imports out of base. [Robert Bradshaw] Changes needed to compile with ts closure plugin. [Robert Bradshaw] Use ttsc and ts-closure-transform plugin. [Robert Bradshaw] Serialization registration to actually get serialization working. [Robert Bradshaw] Container images working on local runner. [Robert Bradshaw] Add a portable job server that proxies the Dataflow backend. (#17189) [Robert Bradshaw] Improvements to dataflow job service for non-Python jobs. [Robert Bradshaw] Get dataflow working. [Robert Bradshaw] User friendly pipeline options. [Robert Bradshaw] Less classes, more functions. [Robert Bradshaw] Add new nullable standard coder. [Robert Bradshaw] Make Apache Rat happy. [Robert Bradshaw] Disable broken codecov. [Robert Bradshaw] Remove last uses of base.ts. [Robert Bradshaw] Remove unneedd file. [Robert Bradshaw] Remove more uneeded/unused files. [Robert Bradshaw] Cleanup tests. [Robert Bradshaw] Minor cleanups to coder tests. [Robert Bradshaw] Addressing issues from the review. [noreply] Apply suggestions from code review. [Robert Bradshaw] Post-merge fixes. [dannymccormick] Delete tags.go [Robert Bradshaw] Update tests to use our actual serialization libraries. [Robert Bradshaw] Another pass at TODOs, removing finished items. [Heejong Lee] [BEAM-14146] Python Streaming job failing to drain with BigQueryIO write [Kenneth Knowles] Add parameter for service account impersonation in GCP credentials [noreply] Merge pull request #17490 from [BEAM-14370] [Website] Add new page about [noreply] [BEAM-14332] Refactored cluster management for Flink on Dataproc [noreply] [BEAM-13988] Update mtime to use time.UnixMilli() calls (#17578) ------------------------------------------ [...truncated 977.15 KB...] opt-einsum==3.3.0 orjson==3.6.8 overrides==6.1.0 packaging==21.3 pandas==1.4.2 parameterized==0.7.5 pbr==5.8.1 pip==22.0.4 pluggy==0.13.1 proto-plus==1.20.3 protobuf==3.20.1 psycopg2-binary==2.9.3 py==1.11.0 pyarrow==7.0.0 pyasn1==0.4.8 pyasn1-modules==0.2.8 pycparser==2.21 pydot==1.4.2 PyHamcrest==1.10.1 pymongo==3.12.3 PyMySQL==1.0.2 pyparsing==2.4.7 pytest==4.6.11 pytest-forked==1.4.0 pytest-timeout==1.4.2 pytest-xdist==1.34.0 python-dateutil==2.8.2 python-snappy==0.6.1 pytz==2022.1 PyYAML==6.0 regex==2022.4.24 requests==2.27.1 requests-mock==1.9.3 requests-oauthlib==1.3.1 rsa==4.8 scikit-learn==1.0.2 scipy==1.8.0 setuptools==58.1.0 six==1.16.0 soupsieve==2.3.2.post1 SQLAlchemy==1.4.36 tenacity==5.1.5 tensorboard==2.8.0 tensorboard-data-server==0.6.1 tensorboard-plugin-wit==1.8.1 tensorflow==2.8.0 tensorflow-io-gcs-filesystem==0.25.0 termcolor==1.1.0 testcontainers==3.5.3 tf-estimator-nightly==2.8.0.dev2021122109 threadpoolctl==3.1.0 tqdm==4.64.0 typing-utils==0.1.0 typing_extensions==4.2.0 uritemplate==4.1.1 urllib3==1.26.9 wcwidth==0.2.5 websocket-client==1.3.2 Werkzeug==2.1.2 wheel==0.37.1 wrapt==1.14.1 zipp==3.8.0 Removing intermediate container b72901e7c69e ---> 5e38482b477a Step 16/31 : COPY target/LICENSE /opt/apache/beam/ ---> 29f1770c9d9a Step 17/31 : COPY target/LICENSE.python /opt/apache/beam/ ---> 5e672b7797c6 Step 18/31 : COPY target/NOTICE /opt/apache/beam/ ---> 99574eb20dc4 Step 19/31 : COPY target/launcher/linux_amd64/boot /opt/apache/beam/ ---> 68f77689a0ea Step 20/31 : ENTRYPOINT ["/opt/apache/beam/boot"] ---> Running in 3ceebd2ffe5f Removing intermediate container 3ceebd2ffe5f ---> 6fb1b37abd9d Step 21/31 : FROM beam as third_party_licenses ---> 6fb1b37abd9d Step 22/31 : ARG pull_licenses ---> Running in 67c1b62ced6c Removing intermediate container 67c1b62ced6c ---> 4b47cc50ab28 Step 23/31 : COPY target/license_scripts /tmp/license_scripts/ ---> eb1ad333502b Step 24/31 : COPY target/LICENSE target/go-licenses/* /opt/apache/beam/third_party_licenses/golang/ ---> 30f43718c7cf Step 25/31 : RUN rm /opt/apache/beam/third_party_licenses/golang/LICENSE ---> Running in 534db125a639 Removing intermediate container 534db125a639 ---> 86736c665e3c Step 26/31 : COPY target/license_scripts /tmp/license_scripts/ ---> 043b128f03fd Step 27/31 : RUN if [ "$pull_licenses" = "true" ] ; then pip install 'pip-licenses<4.0.0' pyyaml tenacity && python /tmp/license_scripts/pull_licenses_py.py ; fi ---> Running in 0bdbbbc306d6 Collecting pip-licenses<4.0.0 Downloading pip_licenses-3.5.4-py3-none-any.whl (17 kB) Requirement already satisfied: pyyaml in /usr/local/lib/python3.9/site-packages (6.0) Requirement already satisfied: tenacity in /usr/local/lib/python3.9/site-packages (5.1.5) Collecting PTable Downloading PTable-0.9.2.tar.gz (31 kB) Preparing metadata (setup.py): started Preparing metadata (setup.py): finished with status 'done' Requirement already satisfied: six>=1.9.0 in /usr/local/lib/python3.9/site-packages (from tenacity) (1.16.0) Building wheels for collected packages: PTable Building wheel for PTable (setup.py): started Building wheel for PTable (setup.py): finished with status 'done' Created wheel for PTable: filename=PTable-0.9.2-py3-none-any.whl size=22925 sha256=a7a36ca4d64c60b4454ad56689adba90e29e2e388f319bcc775b9570636ae475 Stored in directory: /root/.cache/pip/wheels/b8/d5/8b/e0c9765594e0dc8093aae5f67eacc08b9b533da598c710b54a Successfully built PTable Installing collected packages: PTable, pip-licenses Successfully installed PTable-0.9.2 pip-licenses-3.5.4 [91mWARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv [0m[91mINFO:root:Successfully pulled licenses for 134 dependencies [0mSkip pulling license for bs4 Removing intermediate container 0bdbbbc306d6 ---> 99c9384fdecc Step 28/31 : FROM beam ---> 6fb1b37abd9d Step 29/31 : ARG pull_licenses ---> Running in 6356f6f8262d Removing intermediate container 6356f6f8262d ---> ffee725f047b Step 30/31 : COPY --from=third_party_licenses /opt/apache/beam/third_party_licenses /opt/apache/beam/third_party_licenses ---> 5b3608368923 Step 31/31 : RUN if [ "$pull_licenses" != "true" ] ; then rm -rf /opt/apache/beam/third_party_licenses ; fi ---> Running in 85b3f68164bd Removing intermediate container 85b3f68164bd ---> 4c32e8ed6ebf Successfully built 4c32e8ed6ebf Successfully tagged apache/beam_python3.9_sdk:2.40.0.dev > Task :runners:google-cloud-dataflow-java:buildAndPushDockerPythonContainer WARNING: `gcloud docker` will not be supported for Docker client versions above 18.03. As an alternative, use `gcloud auth configure-docker` to configure `docker` to use `gcloud` as a credential helper, then use `docker` as you would for non-GCR registries, e.g. `docker pull gcr.io/project-id/my-image`. Add `--verbosity=error` to silence this warning: `gcloud docker --verbosity=error -- pull gcr.io/project-id/my-image`. See: https://cloud.google.com/container-registry/docs/support/deprecation-notices#gcloud-docker The push refers to repository [us.gcr.io/apache-beam-testing/java-postcommit-it/python] bc1ab5f02dec: Preparing eaa96aa0c339: Preparing 3010b6904b47: Preparing 166b2c7a6ea4: Preparing 5f77aab3bd0d: Preparing e34924c06d73: Preparing 96c5a7b3c74a: Preparing bfd86085cdda: Preparing 1405deb4e2c8: Preparing 60cddfc4dfff: Preparing defbfe42c595: Preparing ffb2f5cb4392: Preparing a7bf081cc35a: Preparing 752f69980d1f: Preparing 3eec322dea55: Preparing 105e55456415: Preparing 7e8c58e9679c: Preparing 9c26176f6534: Preparing 6d95196cbe50: Preparing 5bdcc8e2060c: Preparing 08fa02ce37eb: Preparing a037458de4e0: Preparing e34924c06d73: Waiting bafdbe68e4ae: Preparing a13c519c6361: Preparing 08fa02ce37eb: Waiting bafdbe68e4ae: Waiting a13c519c6361: Waiting 60cddfc4dfff: Waiting defbfe42c595: Waiting 3eec322dea55: Waiting ffb2f5cb4392: Waiting 6d95196cbe50: Waiting a7bf081cc35a: Waiting 105e55456415: Waiting 7e8c58e9679c: Waiting 96c5a7b3c74a: Waiting 1405deb4e2c8: Waiting bfd86085cdda: Waiting a037458de4e0: Waiting 5f77aab3bd0d: Pushed 166b2c7a6ea4: Pushed bc1ab5f02dec: Pushed 3010b6904b47: Pushed eaa96aa0c339: Pushed e34924c06d73: Pushed 96c5a7b3c74a: Pushed 1405deb4e2c8: Pushed 60cddfc4dfff: Pushed defbfe42c595: Pushed 752f69980d1f: Pushed 105e55456415: Layer already exists 7e8c58e9679c: Layer already exists 9c26176f6534: Layer already exists 6d95196cbe50: Layer already exists 5bdcc8e2060c: Layer already exists 08fa02ce37eb: Layer already exists a037458de4e0: Layer already exists 3eec322dea55: Pushed bafdbe68e4ae: Layer already exists a13c519c6361: Layer already exists bfd86085cdda: Pushed ffb2f5cb4392: Pushed The message received from the daemon indicates that the daemon has disappeared. Build request sent: Build{id=062980ec-9238-4d5d-a0d0-52a5e6846cbe, currentDir=<https://ci-beam.apache.org/job/beam_PostCommit_XVR_JavaUsingPython_Dataflow/ws/src}> Attempting to read last messages from the daemon log... Daemon pid: 326966 log file: /home/jenkins/.gradle/daemon/7.4/daemon-326966.out.log ----- Last 20 lines from daemon log file - daemon-326966.out.log ----- eaa96aa0c339: Pushed e34924c06d73: Pushed 96c5a7b3c74a: Pushed 1405deb4e2c8: Pushed 60cddfc4dfff: Pushed defbfe42c595: Pushed 752f69980d1f: Pushed 105e55456415: Layer already exists 7e8c58e9679c: Layer already exists 9c26176f6534: Layer already exists 6d95196cbe50: Layer already exists 5bdcc8e2060c: Layer already exists 08fa02ce37eb: Layer already exists a037458de4e0: Layer already exists 3eec322dea55: Pushed bafdbe68e4ae: Layer already exists a13c519c6361: Layer already exists bfd86085cdda: Pushed ffb2f5cb4392: Pushed Daemon vm is shutting down... The daemon has exited normally or was terminated in response to a user interrupt. ----- End of the daemon log ----- FAILURE: Build failed with an exception. * What went wrong: Gradle build daemon disappeared unexpectedly (it may have been killed or may have crashed) * Try: > Run with --stacktrace option to get the stack trace. > Run with --info or --debug option to get more log output. > Run with --scan to get full insights. * Get more help at https://help.gradle.org Build step 'Invoke Gradle script' changed build result to FAILURE Build step 'Invoke Gradle script' marked build as failure --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
